|
Items in cart [1] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 211
11
Monitoring, Evaluation,
and Continuous Improvement
T
he extent to which programs can meet their objectives is dependent
in part on how they are managed at the national level. The preceding
several chapters outlined the evidence available to characterize the
performance of the Title VI and Fulbright-Hays (Title VI/FH) programs in
each of the eight key areas identified by Congress. However, neither the data
available in the Evaluation of Exchange, Language, International and Area
Studies (EELIAS) database nor the few available evaluation studies enable
an assessment of the relative value of each of the 14 component programs.
Program monitoring and performance measurement efforts have largely
been conducted as a top-down enterprise accompanied by little consultation
with grantees. Yet program performance is also partially dependent on the
extent to which universities and other grantees share program goals and
work collaboratively to accomplish those goals. Although there are a few
examples of mechanisms that facilitate coordination and identification of
promising practices across programs (see the description in Chapter 9 of
CIBERweb), and the center programs have established an effective process
for involvement in national policy making, there is no established process
to facilitate continual assessment and improvement.
This chapter discusses the status of program monitoring and program
evaluation activities and outlines ways that the U.S. Department of Edu-
cation (ED) should advance its efforts in these areas. It also presents the
limitations of current collaborations with grantees and a recommended ap-
proach to continuous improvement that would complement and potentially
enhance the department’s efforts to ensure the effectiveness of the Title
VI/FH programs.
OCR for page 212
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
PROGRAM MONITORING
At ED, program monitoring is carried out by program officers in the
International Education Programs Service (IEPS) who are assigned to a par-
ticular program(s) or, in the case of the National Resource Center (NRC)
program, a particular world area. Unlike in the past, when financial and
project (programmatic) monitoring was conducted by separate staff, pro-
gram officers are currently responsible for both functions, although their
work focuses primarily on financial monitoring. At the same time, the
number of program staff required to administer has increased over time. In
recent years, full-time-equivalent staffing levels have decreased, dropping
from 23 in FY 2003 to 21 in FY 2006, despite congressional concerns ar-
ticulated in the FY 2003 budget.1
Limited staff combined with limited travel resources and departmental
policies related to travel approval have severely limited the ability of staff
to visit grantees. ED has an increased emphasis in project-level evaluation
in part to address the limits on their ability to conduct on-site reviews.
For example, the number of points awarded for evaluation plans has been
increased in the past several NRC competitions, and it is now the criterion
with the highest number of points (25 of 165). In addition, ED is now
requiring NRCs to include a professional evaluator as well as a peer evalu-
ator in their evaluation plans. The effectiveness of this new emphasis on
evaluation is as yet unknown.
Like most federal programs, the Title VI/FH programs require their
grantees to submit annual reports to review the progress of individual
grantees and to assist efforts to evaluate the performance of the programs
overall. For most of the programs’ history, annual reports were submitted
in paper format. In recent years, as discussed below, IEPS shifted from a
paper to an electronic, web-based system (EELIAS) for collecting grantee
information.2
In 1978, the General Accounting Office recommended that Title VI/FH
programs share evaluative data with grantees to provide information about
accomplishments and guidance for possible improvements. At the request of
the President’s Commission on Foreign Language and International Studies
in 1979, ED staff prepared an analysis on how NRCs use their funds based
on reports submitted by grantees (Scheider, 1979). Other staff-generated
memos to the NRC and Foreign Language and Area Studies (FLAS) Fel-
lowships directors distributed in the mid-1990s provide analytic syntheses
1 Congress stated: “The conferees are disappointed that the Department has not fully ad-
dressed the staffing needs of the Title VI and Fulbright-Hays international education pro-
grams” (House Report 108-010).
2 Shortly before release of the committee’s report, the system was revised and released as the
International Resource Information System.
OCR for page 213
MONITORING, EVALUATION, AND CONTINUOUS IMPROVEMENT
BOX 11-1
Categories of Performance Measures
Inputs: Resources used (e.g., amount of funding).
Activities: Types of work performed (e.g., number of publications or workshops,
number of students who study abroad).
Outputs: Results from program activities (e.g., number of students enrolled).
Outcomes: An accomplishment attributable to program outputs (e.g., improve-
ment in language proficiency, job placement).
Impact: Achievement of broad social objectives (e.g., increased global
understanding).
SOURCES: Adapted from Joyner (2006a).
of data from annual reports on such criteria as average number of degrees
awarded, career choice, and distribution of degrees by discipline (Schneider,
1995). The committee was told that data had been compiled from reports
at the end of every funding cycle since the 1970s. Other reports on program
performance, such as those mentioned above, seem to have disappeared
with the transition to an electronic system.
Performance Measures
The Government Performance and Results Act (GPRA) requires all
federal programs to report measures of program performance, with several
types of possible measures (see Box 11-1). Until recently, ED had perfor-
mance measures for the NRC and FLAS programs only, which were used
to report performance on all the Title VI domestic programs. After an Of-
fice of Management and Budget (OMB) review under the Program Assess-
ment Rating Tool (PART) process3 that resulted in a rating of “results not
demonstrated,” the program developed specific performance measures for
each of the 14 programs (see Table 11-1). There are now two performance
measures and one efficiency measure for most programs with an emphasis
on measures intended to indicate outcomes, as preferred by OMB. The
measures were recently approved by OMB.
3 OMB conducts periodic program reviews using PART and assigns one of five ratings as a
result of the process: (1) effective, (2) moderately effective, (3) adequate, (4) ineffective, and
(5) results not demonstrated.
OCR for page 214
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
TABLE 11-1 Planned Title VI/Fulbright-Hays Performance Measures
Program Performance Measures Efficiency Measure
NRC Percentage of critical languages Percentage of Cost per master's
taught, as reflected in the list of employed master's or doctoral
critical languages referenced in and doctoral degree degree graduate
the Title VI program statute. graduates in fields employed in fields
of postsecondary of postsecondary
education or education or
government. government.
FLAS Average language competency Percentage of Cost per fellow
score of FLAS fellowship employed master's increasing average
recipients at the end of one full and doctoral degree language competency
year of instruction will be at graduates in fields by at least one level.
least 1.20 levels higher than their of postsecondary
average score at the beginning of education or
the year. government.
CIBE Percentage of graduates Percentage of Cost per master’s,
of a Ph.D. or master’s, projects reported including MBA,
including MBA, program and validated as degree recipient
with significant international of high quality or Ph.D. graduate
business concentration at the or successfully employed in
postsecondary institution who completed. business-related
are employed in business-related fields, including
fields, including teaching at a teaching in a business
business school. school.
IRS Number of outreach activities Percentage of Cost per high-
that result in adoption or further projects reported quality, successfully
dissemination within a year, and validated as completed project.
divided by the total number of of high quality
IRS outreach activities conducted or successfully
in the current reporting period. completed.
LRC Number of outreach activities Percentage of Cost per high-
that result in adoption or further projects reported quality, successfully
dissemination within a year, and validated as completed project.
divided by the total number of high quality
of LRC outreach activities or successfully
conducted in the current completed.
reporting period.
UISFL Percentage of critical languages Percentage of Cost per high-
addressed/covered by foreign projects reported quality, successfully
language major, minor, or and validated as completed project.
certificate programs created of high quality
or enhanced; or by language or successfully
courses created or enhanced; or completed.
by faculty or instructor positions
created with UISFL or matching
funds in the reporting period.
OCR for page 215
MONITORING, EVALUATION, AND CONTINUOUS IMPROVEMENT
TABLE 11-1 Continued
Program Performance Measures Efficiency Measure
BIE Number of outreach activities Percentage of Cost per high-
that result in adoption or further projects reported quality, successfully
dissemination within a year, and validated as completed project.
divided by the total number of of high quality
BIE outreach activities conducted or successfully
in the current reporting period. completed.
TICFIA Percentage of projects reported N/A Cost per high-
and validated as of high quality quality, successfully
or successfully completed. completed project.
AORC Percentage of projects reported Percentage of Cost per high-
and validated as of high quality scholars who quality, successfully
or successfully completed. indicated they completed project.
were “highly
satisfied” with the
services the center
provided.
DDRA Average fellow increases Percentage of Cost per grantee
language competency by at least projects reported increasing language
0.75 level. and validated as competency by at
of high quality least one level in one
or successfully area (or all three).
completed.
FRA Average fellow increases Percentage of Cost per grantee
language competency by at least projects reported increasing language
0.50 level. and validated as competency by at
of high quality least one level in one
or successfully area (or all three).
completed.
GPA Average fellow increases Percentage of Cost per grantee
language competency by at least projects reported increasing language
0.50 level. and validated as competency by at
of high quality least one level in one
or successfully area (or all three).
completed.
SA Percentage of projects reported N/A Cost per high-
and validated as of high quality quality, successfully
or successfully completed. completed project.
IIPP Percentage of employed IIPP Percentage of IIPP Cost per IIPP
graduates in government or participants who graduate employed
international service. complete a master’s in government or
degree within six international service.
years of enrolling
in the program.
OCR for page 216
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
The measures are based on a combination of quantitative (e.g., gradu-
ate placements, language courses, self-reported language proficiency) and
narrative (e.g., accomplishments) information provided by grantees to ED
via its web-based grantee reporting system.
The performance measures identified appear to have been developed
with little or no input from the universities or other organizations who
receive funding. Although designed to measure outcomes as much as pos-
sible, little thought seems to have been given to the goals of the programs
or how to best measure performance against those goals. Several universi-
ties expressed concern about the measures for which they are being held
accountable, pointing to several issues: (1) Monitoring the percentage of
languages taught overlooks the level of language taught and might create
a disincentive to offer advanced-level language courses. (2) Placements in
fields other than academia and government are reasonable outcomes not
reflected in the placement measures (see Chapter 6 for further discussion
of this issue). (3) The two measures used do not reflect the emphasis placed
on outreach.
EELIAS Database
The EELIAS system, reportedly developed with input from the field, ap-
pears to have been designed to capture all potentially useful data, using the
NRC program as the model (see Box 11-2 for background information). It
is not clear, however, whether full consideration was given to how the data
would be used for program monitoring—that is, the specific performance
measures to be used and how the required data would be used to report
those measures. In practice, the system has been used by ED exclusively for
two purposes: (1) to enable project officers to review the performance of
individual grantees and make decisions about annual continuation grants
for multiyear grants4 and (2) to report performance measures. The project
officer review uses primarily narrative information and appears to be a one-
way review, with no feedback to the grantee.
The purpose of the extensive quantifiable data collected that are not
used for performance measures is also unclear. No information has been
given to grantees on how their performance compares with other grantees,
and no other aggregate reports have been produced. The system is widely
viewed by universities as a burdensome, time-consuming requirement, not
a resource. Many universities pointed out that the time required to collect
and then enter the required information is significant, particularly in rela-
4 The committee was told that a continuation award has never been denied as a result of
this review.
OCR for page 217
MONITORING, EVALUATION, AND CONTINUOUS IMPROVEMENT
BOX 11-2
Evolution of ED’s Grant Monitoring Database
The Evaluation of Exchange, Language, and International Area Studies
(EELIAS) database was developed by the National Foreign Language Center
under a grant from the Title VI International Research and Studies program.
Developed over a period of several years, it was intended to be used as a grant
monitoring tool and to provide data for ED to report program performance to the
Office of Management and Budget, as required by the Government Performance
and Results Act. It was developed as an online grantee reporting system intended
to provide information on the language, international, and area studies compo-
nents of the Title VI/FH programs. National Resource Centers (NRC) and Foreign
Language and Area Studies (FLAS) grantees were the first to report information
into the system, beginning in the early 2000s. Other programs were phased in
over time, and an ED contractor has been entering historical grantee information.
Multiyear grantees submit both annual and final reports. According to both ED
staff and the committee’s own review, the data are most reliable for the NRC and
FLAS programs. In fact, there is some thought among other grantees that the
system was designed for those programs and imposed onto their programs.
Technical issues with the design of the system, as well as incompleteness
of data, make it difficult to use aggregate data for program evaluation purposes.
ED worked with a contractor to develop a revised system designed to correct
technical issues. The revised and renamed International Resource Information
System should address some of the database’s technical limitations and make
grantee-level information more readily accessible. However, it is unclear whether
and if so, how, the new system will result in improved program-level or aggregate
information or public access to data used in reporting performance measures,
which are necessary to ensure transparency.
tion to the relatively small amount of money received and given that they
do not get any summary data or other feedback.5
The department has grappled with the same technical issues related
to lack of internal controls, frequent use of open-ended questions and an
“other” category, limited data verification, and missing data encountered
by the committee (see Appendix B, Box B-1) that limited our ability to use
much of the EELIAS data for program-level review. The committee notes
that the system was designed under a grant rather than a contract. Grants
do not typically allow significant feedback from the sponsoring agency
and are not the ideal mechanism for developing a program monitoring
5 In
fact, several universities reported that they were not aware that they could access their
own data, and certainly did not know how to do so once it had been submitted.
OCR for page 218
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
tool. Nonetheless, ED made the decision to implement the system across
all Title VI/FH programs. Perhaps recognizing the limitations posed by the
grant mechanism, ED is now managing the system with the assistance of
a contractor, which enables substantially greater control over the product.
Concurrent with the committee’s deliberations, ED was in the midst of a
systematic review of the database, with the assistance of its contractor. ED
is aware of many of the issues encountered by the committee. In fact, it
independently shared observations about the system’s limitations and its
proposed solutions. Near the end of the committee’s deliberations, ED pub-
lished in the Federal Register proposed system changes that will reportedly
fix most if not all of the technical issues with the system we have identi-
fied (U.S. Department of Education, 2006d). Select grantees were asked to
review the navigability of the new system, including a prototype for a new
user interface.
However, it appears that use of the redesigned system, which was re-
named the International Resource Information System when launched as
a “new” system (see http://www.ieps-iris.org) shortly before release of the
committee’s report, will continue to focus on individual grant monitoring
and reporting performance measures. The redesign will make access to se-
lect system components more readily available, but it will not make basic
data available. It also does not make data used for performance measures
readily accessible. In addition, the redesign does not appear to address ques-
tions related to the purpose and use of the required data.
Data Transparency
At the time of this writing, although ED reported that it has considered
open web-based access to the database, a web-based interface with addi-
tional query potential (e.g., languages taught, graduate placements), and
production of additional summary reports, a decision had not been made
about whether to make data public and, if so, which data. Several univer-
sities made the observation that open access to the data would reveal any
differences in the way universities are reporting data and highlight missing
data, informally encouraging data continuity. Greater system transparency
would be likely to facilitate a vested interest on the part of grantees in the
timeliness and comparability of data.
The committee is aware of only three efforts (two of which were
requests for the same data) by organizations other than the system devel-
oper6 to access the data. The Social Science Research Council planned to
undertake an evaluation of Middle East NRCs funded by the International
6 The system developers conducted two related studies that used EELIAS data to review
language enrollments: Brecht and Rivers (2000) and Brecht et al. (2007).
OCR for page 219
MONITORING, EVALUATION, AND CONTINUOUS IMPROVEMENT
Research and Studies Program (IRS) using EELIAS data and was required
to submit a Freedom of Information Act request to access the database.
However, such issues as the difficulty of analyzing responses to open-ended
questions related to languages taught and type of instructor hampered the
ability of the research team to use the data to answer their initial questions
(Browne, 2006). In addition, Steven Heydemann of Georgetown Univer-
sity and Martin Kramer of the Washington Institute for Near East Policy7
submitted separate requests to ED and received a spreadsheet with NRC
placement data for 2002 (see the discussion in Chapter 6). Although using
the same data, they came to very different conclusions about the program’s
success in producing graduates who enter government service, based on
their starting assumptions. Had these data been publicly available, a more
informed public discourse might have been possible.
If the planned improvements to the database are well implemented, the
system may offer the potential to provide better information to the field.
Well-designed grantee data systems can be useful program monitoring tools
to help track trends, collect data on outputs, and ensure that funds are used
for their intended purposes. When the information collected is viewed as
useful to those reporting it, and information is in fact returned to them,
it is likely to be of higher quality. It is important to note that the planned
improvements to the system are aimed at fixing technical glitches and
improving user navigability. Consideration was not given to whether the
data collected should be collected and how it would be used for program
monitoring or improvement purposes. Basic functionality will not change.
That is, data elements are not being increased or decreased. ED might be
well served by convening grantees to discuss the current data collected
and how it can be modified to address mutual objectives (see below for
further discussion). ED should undertake a systematic evaluation involving
grantees to ensure that the changes made through the launch of its redesign
system have been implemented effectively, that the data are fully usable
and understandable, and that the expected value of the data is clear. In
comments shared with the committee, Brustein (2006) reinforced this and
related points, claiming that there is insufficient dissemination of project
outcomes and best practices, as well as a lack of accessible and searchable
databases.
PROGRAM EVALUATION
Although well-designed grantee data systems are vital program moni-
toring tools, they are rarely adequate to assess the effectiveness of programs
7 MartinKramer requested the data to verify information included in an article published by
Steven Heydemann based in part on the placement data.
OCR for page 220
0 INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
in terms of outcomes and impacts or to understand why identified trends
are occurring. These more detailed and nuanced assessments require well-
designed program evaluations that include information on both what is
achieved and how it is achieved. Few Title VI/FH program evaluations have
been conducted in general; even fewer have focused on program outcomes.
Of the evaluations that have been conducted (see Appendix B, Attachment
B-1), most have involved surveys of grantees only. Although control groups
are infeasible, almost none have had a comparison group. Many have been
funded by the IRS Program via grants designed and implemented with little
ED input. Although IRS proposals are reviewed by peer review panels, and
some of the criteria seem to be aimed at ensuring a quality evaluation proj-
ect (e.g., 15 of 100 points for an evaluation plan, 10 points for adequacy of
method and scope of project, 10 points for knowledge to be gained), grants
are initiated and managed by the principal investigator. Grants are typically
not the ideal mechanism for federal program evaluation, given the relative
independence given to grantees. Contracts, which allow more specification
of research questions and methodology and greater involvement in the re-
search design and implementation, would be a preferable mechanism.
Only one recent evaluation—the survey of fellowship recipients that in-
cluded the FLAS and Doctoral Dissertation Research Abroad (DDRA) pro-
grams in addition to two other ED fellowship programs—was conducted
under an ED contract that enabled regular, ongoing review and feedback
by the department. This survey was funded by the Office of Planning,
Evaluation and Policy Development (OPEPD). It is also the only evaluation
that included more than one Title VI program. As Gabara (2006) reported
to the committee, more research should be done on national needs, and
evaluations of programs should look at their accomplishments over the
past 15 years. Given that the programs are now firmly established, more
periodic, well-designed evaluations with clearly articulated outcomes are
warranted—perhaps every four to five years. This would make available
up-to-date program evaluation information for PART reviews and for re-
authorization deliberations.8 Christian (2006) reported that there seems to
be some dissatisfaction with the way projects are evaluated (regarding the
indicators used), as well as some thought that it is difficult to measure suc-
cess for some of them, or that they cannot all be evaluated in the same way.
Program evaluations, ideally with input from the field, will have to carefully
consider the expectations of the programs and the use of best practices in
achieving those goals.
8 Under PART, each program is to be reviewed at least every five years. The Higher Education
Act is supposed to be reauthorized on a six-year cycle.
OCR for page 221
MONITORING, EVALUATION, AND CONTINUOUS IMPROVEMENT
CONTINUOUS IMPROVEMENT
Modern governance theory suggests that educational programs are
more likely to improve if goals are clear, participants have a voice in defin-
ing objectives and internalizing the goals in their own systems, quantitative
indicators of process are available and made public, and efforts are made to
identify and duplicate promising practices (see, for example, Liebman and
Sabel, 2003). Neither ED nor the established mechanisms to communicate
across grantees have made this a priority. In fact, universities (the primary
Title VI/FH grantees) have been little involved in ED’s efforts to monitor
performance or identify strategic directions, although there have been some
recent attempts to improve in this area.
ED has historically convened a meeting in Washington, DC, of all
grantees for each program after each competition, but it has not held na-
tional meetings in the years between the multiyear award cycles and has met
separately with each program. In September 2006, ED convened all NRC
and Language Resource Centers (LRC) grantees at the same event, the first
such meeting involving leaders of both programs. The agenda included not
only presentations by IEPS staff about grant reporting requirements, but
also panels of invited experts and grantees, who shared their knowledge of
such complex topics as assessing language acquisition and evaluating the
impact of study abroad. This event also included an open town hall meet-
ing for feedback from the field to IEPS staff. ED plans to convene more
information-sharing and technical assistance meetings in 2007: the first will
focus on language acquisition, and the second will be about outreach. This
appears to represent a distinct shift from earlier years and an attempt to
actively engage the field in relevant issues.
Each of the major programs has established an independent mecha-
nism for communication across projects. Each program has a dues-paying
organization of program directors: NRCs have the Council of NRC Direc-
tors, LRCs have the Council of Directors of National Foreign Language
Resource Centers, and Centers for International Business Education and
Research (CIBER) have the Association for International Business and Re-
search. At a minimum, each group gets together (without ED staff present)
in conjunction with the ED-sponsored grantee meeting for its program. The
Council of NRC Directors organized a major conference at the University
of California, Los Angeles, in 1997 prior to renewal of the Higher Educa-
tion Act. The conference, which included grantees of other Title VI pro-
grams as well as NRC directors, was reported to have a major influence on
Congress and to have led to creation of the Technological Innovation and
Cooperation for Foreign Information Access (TICFIA) program (Ruther,
2003). American Overseas Research Centers (AORC) have created the
Council of American Overseas Research Centers.
OCR for page 222
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
The LRCs and the CIBERs also have websites (the content of the
CIBER web site is significantly greater than the LRC site; see Chapter 9)
and collect and synthesize performance data based on the data submitted
via EELIAS. IEPS staff have actively encouraged the creation and content of
the CIBER web site and have supported a shared web portal for the Busi-
ness and International Education (BIE) program, hosted by Grand Valley
State University (2006). The committee was told that the Council of NRC
Directors agreed in September 2006 to establish a web presence, but that it
is on hold pending a decision by ED regarding what EELIAS information
they might be making available via the web.
The director of each council or association is also a member of the
Coalition for International Education (CIE), whose membership includes
a range of other national organizations concerned with higher education,
foreign language, or international education issues. CIE aims to build
consensus within the higher education community on policies affecting the
programs and advocates for grantees in the annual federal budgeting and
appropriations process. CIE convened a national conference in 2003, with
support from ED and the Ford Foundation, for Title VI grantees to discuss
demand for global education after the terrorist attacks of September 11,
2001, and to prepare for reauthorization of the Higher Education Act. The
upcoming 50th anniversary of Title VI in 2008 is likely to provide another
opportunity for a reflective view of the programs.
In addition to sharing information through these national associa-
tions, grantees report that they discuss common issues and opportunities
for collaboration through informal meetings at professional associations,
such as the Asia Studies Association and the Joint National Committee on
Languages.
These efforts have an important role to play. However, few of these
efforts are aimed at collaborations across programs or between ED and
grantees, efforts that would encourage a partnership approach to the pro-
grams. They are also not being implemented in a coordinated, systematic
manner that would facilitate accomplishment of common goals. ED appears
to recognize the value of collaboration, as illustrated in its introduction of
an invitational priority related to collaboration in the NRC and LRC 2005
competitions. Curiously, however, the NRC priority included collaboration
with LRC, AORC, and CIBER programs but did not mention other NRCs.
In addition, the collaborations are aimed at interactions between grantees,
with little department involvement and no attention to the way the program
is managed at a national level or to the content of collaborations.
Several National Academies reports in the health care arena have called
for a collaborative approach to performance measurement that embraces
an interactive process involving the public and private sectors in the de-
velopment of program goals, performance measures consistent with those
OCR for page 223
MONITORING, EVALUATION, AND CONTINUOUS IMPROVEMENT
goals, and evaluation of the measures (see National Academy of Sciences,
National Academy of Engineering, Institute of Medicine, 2007; Institute of
Medicine, 1997). The recent report of the secretary of education’s Com-
mission on Higher Education (U.S. Department of Education, 2006a) re-
inforced a similar idea when it called for a change to a system based on
performance, emphasizing that higher education institutions need to “em-
brace and implement serious accountability measures.”
A collaborative approach in which ED and the universities work to-
gether to develop performance measures should increase the programs’ ef-
fectiveness and encourage continuous improvement. Universities are more
likely to internalize performance measures if they have had a voice in their
creation. Such measures can also help universities learn from each other.
Once specific performance measures are identified, universities can
compare their performance on these measures with those of their peers.
This could lead to the identification of promising practices that they may
be able to adapt. For example, why is a particular university or program
particularly good at increasing language proficiency? How has a given
program been able to establish linkages with their college of education
or local education authority? Why are graduates of a particular program
getting significant numbers of government jobs? If universities find that
these results are brought about by practices that are transferable, they are
likely to improve their own performance. Systems to encourage and sup-
port measurement of performance and sharing of information concerning
promising and transferable approaches could improve overall performance
of the Title VI/FH programs.
AWARD TRANSPARENCY
At the most basic level, successful grant applications are a source of
information about best practice; at a minimum, they represent proposals
that expert review panels rated as most responsive to application criteria.
During site visits and the committee’s meeting with directors of newly
awarded NRCs, multiple project staff mentioned the potential usefulness of
applications as a source of information about what it takes to put together a
successful program. Feedback on applications was also cited as a potentially
useful source of information about program strengths and needed program
improvements, even for successful applicants.
At the same time, the committee heard repeated concerns about the
difficulty of accessing successful grant applications. According to ED staff
and grantees, applications can only be viewed (but not removed or copied)
in the ED offices in Washington, DC. This limits public access generally
and creates a clear disadvantage for anyone not based in the area. Limiting
access to successful applications may limit unsuccessful applicants’ ability
OCR for page 224
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
to identify shortcomings and develop more competitive proposals in future
competitions, thus limiting competition. In addition, since they are federal
programs, the process used to make Title VI/FH awards and the results
of the competition should be readily available to the public. Information
technology makes this easy to do.
The committee also heard specific criticisms about the grant selection
process, which has moved to electronic review. In the past, reviewers met
face-to-face to discuss their review and their planned point assignment for
each criterion. Ratings are now done electronically, with a teleconference
held for reviewers to discuss their ratings. Critics claim that this change has
made it difficult to deliberate and has reduced substantive comment on ap-
plications. Wiley and Schneider (2006) state that in the case of IRS grants,
for example, the effectiveness of the program has been negatively impacted
by the grant selection procedure. The committee notes, however, that the
electronic review process helps minimize travel costs for reviewers and is
designed to increase review efficiency. Curiously, while electronic review
should make sharing of review results with applicants a quick process, ED
prints and mails paper copies of the review.
CONCLUSIONS AND RECOMMENDATIONS
Monitoring, evaluation, and continuous improvement efforts at ED
have been affected by staffing limitations, the availability of data, resources,
and program leadership. The committee was told of several initiatives un-
der way that will improve recent efforts, including systematic annual grant
review processes, redesigning the data system, and plans for future grantee
meetings focused on issues important to the field. The grantees themselves
have also implemented mechanisms to improve collaboration within and
among programs. Nonetheless, there is much room for continued improve-
ment both within ED and between ED and its grantees.
Program Monitoring
The limitations and burden of the ED data system are widely recog-
nized, including by ED. The recently implemented redesign appears to be
aimed at addressing the technical issues encountered by the committee.
However, the usefulness of the system will not be fully realized unless the
grantees gain some advantage from the demanding data entry required.
Universities should be able to compare their performance with that of other
similar grantees. The redesigned system should be used more aggressively
to report program performance beyond the narrow criteria reflected in the
performance indicators, and it should include the monitoring of trends.
OCR for page 225
MONITORING, EVALUATION, AND CONTINUOUS IMPROVEMENT
Finally, data should be publicly available to facilitate open, public discourse
about the program and its accomplishments.
Conclusion: The original online data reporting system for Title VI and
Fulbright-Hays programs (EELIAS) is inadequate, is difficult to use,
and has significant consistency problems as well as a lack of transpar-
ency in the data collected.
Recommendation 11.1: The Department of Education should ensure
that its new data system, the International Resource Information Sys-
tem, provides greater standardization, allows comparison across years
and across programs, and provides information to all grantees and to
the public.
Program Evaluation
Few program evaluations have been conducted; those that are available
have been generated mainly by interested researchers, conducted as grants,
and covered a single program. Even the ED-funded evaluation of FLAS and
DDRA recipients did not emanate from the program office.
Conclusion: At the present time, limited information is available to
rigorously assess the outcomes and impacts of the Title VI and Ful-
bright-Hays programs and the nature of the funding (partial funding
of a larger set of activities) makes it difficult to assess outcomes and
impacts.
Recommendation 11.2: The Department of Education should com-
mission independent outcome and impact evaluations of all programs
every 4 to 5 years.
Well-designed program evaluations would be a more reliable approach
to determine program outcomes and impacts than use of performance mea-
sures. There are several options available to fund evaluations. First, IEPS
could develop evaluations in collaboration with OPEPD, as it did for the
survey of fellowship programs. Another alternative would be to fund evalu-
ations using the 1 percent of the appropriation available for program evalu-
ation, national outreach, and information dissemination. However, these
are the same resources currently being used for redesign and maintenance
of the IEPS data system, and to date they have not been used for program
evaluation. Although program evaluations have been funded through IRS
in the past, this mechanism provides little opportunity for national-level
direction or guidance.
OCR for page 226
INTERNATIONAL EDUCATION AND FOREIGN LANGUAGES
Continuous Improvement
To date, neither the Title VI/FH programs nor the universities that
make up the bulk of its grantees have established a process that facilitates
continuous improvement. ED has by and large implemented its reporting
system and performance measures using a top-down approach, with little
buy-in from grantees or consideration of data collection costs, and no clear
rationale for the measures chosen. Networks created by grantees provide
a framework for interactions across grantees, but there is large variation
in the role of the networks, and they function largely independent of ED.
There is little activity that supports interactive governance involving col-
laborative specification of program goals, development of performance
measures, or assessment of the extent to which goals are accomplished.
Recommendation 11.3: The Department of Education should work
with universities to create a system of continuous improvement for
Title VI and Fulbright-Hays programs. The system would help de-
velop performance indicators and other improvement tools and should
include networks of similar centers (National Resource Centers, Lan-
guage Resource Centers, Centers for International Business Education
and Research) and university officials with overall responsibilities in
language, area, and international studies.
While the system could build on existing networks, the process de-
veloped should include senior university officials with university-wide
decision-making authority that affects language, area, and international
studies. Such officials will have the ability to speak on behalf of the broader
pressures and opportunities affecting the programs and reflect on how the
discussions regarding the Title VI/FH programs would be affected or would
affect the larger university system. The system should also capitalize on
existing expertise in universities on given world areas or languages. As part
of this process, ED might consider convening individuals with established
credentials in foreign languages and cultures in major world areas to build
on their successes.
Award Transparency
The most basic example of a successful practice is a successful grant ap-
plication. Yet unlike some other federal agencies, ED has not made applica-
tions readily available. This lack of access hampers competition and thwarts
public access to information about a federally funded program. Although
a successful application does not necessarily indicate that a project will be
successful or well implemented, sharing successful applications provides
OCR for page 227
MONITORING, EVALUATION, AND CONTINUOUS IMPROVEMENT
one public resource on which to build other efforts aimed at continuous
improvement.
Conclusion: Sharing successful grant applications could improve future
competitions and contribute to a continual improvement process.
Recommendation 11.4: The Department of Education should make its
award selection process more transparent, including making successful
applications publicly available via the Internet.
ED has been moving all of its grant competitions to the web-based
http://resource-grants.gov. Although this web-based system is not currently
used to make applications publicly available, it has the capacity to do so. It
is difficult to imagine an applicant who does not prepare his or her applica-
tion in electronic form. Submitting it electronically is a logical next step.
ED could establish a process for making the applications publicly available,
while protecting any necessary financial or other information.
An alternative model for an electronic system, Fastlane, is used by
all programs at the National Science Foundation (NSF).9 Fastlane can be
used to conduct virtually all business with NSF, including submitting grant
proposals, reviewing grant proposals, and determining the status of one’s
proposal, including comments from reviewers. It does not make unsuc-
cessful applications or rankings or comments about individual proposals
available to the public or to anyone other than the applicant. All successful
applications, however, are readily available.
Of course, none of the recommendations in this chapter can be effec-
tively implemented without an institutional structure in the department that
supports innovation, recognizes the importance of strong program leader-
ship, and encourages program change. The next chapter turns to discussion
of these issues.
9 See https://www.fastlane.nsf.gov/fastlane.jsp.
