Rebuilding the Research Capacity at HUD (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

8 Dissemination As is clear from the previous chapters, PD&R produces a great deal of both internally generated and externally funded research that can not only inform the development of the department’s policies but can also be of considerable value to various external constituencies, such as the hous- ing and housing finance industries, housing advocates, and local public officials. How well this potential is realized depends in part on the quality and relevance of this research, as addressed elsewhere in this report. But it also depends on how effective PD&R is in making the research available to external audiences in a timely and user-friendly manner. Dissemination is a challenge in and of itself, given both the large volume of materials produced annually, the constantly changing technology that requires mastery, and the wide range of purposes they would ideally serve. Each of the constituencies mentioned above has different interests, with respect to both the substance of research and the presentation of results. A comprehensive assessment of PD&R’s information dissemination activities was not a key part of the committee’s charge. However, the com- mittee did examine in some depth the clearinghouse, HUD USER, managed by the Research Utilization Division, which constitutes PD&R’s primary information dissemination activity. The committee had available to it two fairly recent independent assessments, commissioned by PD&R, of various aspects of PD&R’s product dissemination activities. Members of the com- mittee also have considerable collective personal experience as consumers of PD&R-generated information and familiarity with the information dis- semination activities of other research organizations. The remainder of this chapter describes the various dissemination elements, provides a committee 139 

140 REBUILDING THE RESEARCH CAPACITY AT HUD perspective on HUD USER and offers some conclusions and recommenda- tions for the future. DISSEMINATION ACTIVITIES PD&R’s dissemination activities take a number of forms. It publishes three periodicals, one of which is devoted entirely to publicizing its research and one of which is partly devoted to that purpose; it maintains a listserv; and it publishes research studies in both hard copy and electronically via its own website. HUD USER PD&R’s primary dissemination vehicle is its information clearinghouse, HUD USER, which was established in 1978. It publishes and markets PD&R’s research reports and other documents. Publications are announced primarily through Research Works, a monthly newsletter, and the listserv, which has been maintained since 1997. A small number of publications are announced through press releases, which are mainly aimed at the trade press. PD&R Publications PD&R’s three publications are Research Works (successor to Recent Research Results), a monthly newsletter summarizing several recent pub- lications in each issue; CityScape: A Journal of Policy Development and Research, a scholarly journal that has published 24 issues on an irregular schedule since it was founded in 1994; and U.S. Housing Market Condi- tions, a quarterly report also founded in 1994. The last publication contains statistics on the national housing and mortgage markets, and reports on national, regional, and local housing markets; it is produced by the Office of Economic Affairs and draws on work by PD&R’s field economists as well as headquarters staff. U.S. Housing Market Conditions is a very useful source of information: much of the information is compiled and published by other government agencies or private entities, but it is helpful to have it in a single source, and the format is convenient. In addition, issues typically contain a short article reporting on some quantitative internal research or program support activity. Recent issues include reports on the performance of government sponsored enterprises (GSEs) under HUD’s home purchase goals (Novem- ber 2007), use of the American Community Survey for estimating income limits and fair market rents to be used in HUD programs (August 2007), and tabulations of first-time home buyers from the 2005 American Hous- 

DISSEMINATION 141 ing Survey ([AHS] May 2007). These interesting articles are not apparently published elsewhere. CityScape includes both internal and external research. Some issues have been devoted in whole or in part to specific topics, occasionally serv- ing as a medium of publication for PD&R-sponsored conferences. One issue (Volume 8, No. 1, published in 2005) was composed largely of studies conducted by PD&R’s research cadre of independent scholars on various aspects of HUD’s subsidized housing programs. Another issue (Volume 7, No. 1, published in 2004) was produced by the Office of University Partner- ships (OUP) and consisted of articles based on the doctoral dissertations of students supported through OUP. Outreach to Constituencies and Policy Makers Individual studies are made public in three or four forms. There is the full study itself, usually published in hard copy and made available electronically on HUD USER. When the study is accessed, the first screen consists of a paragraph or two describing the study, but not usually report- ing any of the results. The study can be accessed from this screen. Studies almost invariably include an executive summary as well as the text. Finally, a typical issue of Research Works summarizes four recent studies, in articles of about 800-1,000 words each. ASSESSMENT OF HUD USER To its credit, PD&R has commissioned two independent reports on its product dissemination activities: an overall assessment of the usefulness of its products, and a subsequent assessment of its website (Vreeke et al., 2001; Bansal et al., 2005). These complement each other. The earlier overall assessment analyzed both purchases and downloads of PD&R publications, as well as citations to PD&R publications in the professional literature. It concluded that customers were generally satisfied with the documents themselves, but identified several ways to improve dissemination. It also provided some information on the demand for reports issued between 1995 and 2000. This analysis was limited to the most popular reports over that period, which were mainly either the Urban Policy Reports issued bienni- ally by HUD or publications on building technology. There was an upward trend in orders for these reports. The overall assessment analyzed dissemination of reports issued between 1995 and 2000. It found an upward trend in orders for the most popular reports over that period: They were mostly either the Urban Policy Reports issued biennially by HUD or publications on building technology. 

142 REBUILDING THE RESEARCH CAPACITY AT HUD Downloading documents from the web was much more popular than ordering hard copies through Research Works or HUD USER itself. A prime example is Creating Defensible Space, written by architect Oscar Newman and published in 1996, describing physical design approaches to preventing crime in urban neighborhoods. This publication sold about 2000 hard copies through 2000, about 400-500 per year; it was downloaded from the website 14,000 times in 2000 alone. Overall, there were more than 600,000 downloads from the HUD USER website between December 1999 and November 2000. (PD&R was unable to track downloads by product or user before December 1999.) Documents based on PD&R data collection or analysis constituted a large share of these downloads; proposed and final rules for fair market rents alone accounted for more than a quarter, and the list of qualified census tracts for the Low-Income Housing Tax Credit (LIHTC) Program was also among the most popular. The assessment also reports that there were more than 84,000 downloads for the 20 most popular PD&R research reports. Unfortunately, it does not include a total for all PD&R documents combined. The most popular were the Residential Rehabilitation Inspec- tion Guide (National Institute of Building Sciences, 2000) and Creating Defensible Space (Newman, 1996), followed by several other reports on technology, the initial report on the Moving to Opportunity (MTO) for Fair Housing demonstration, and the 1997 AHS. The popularity of publications varied by the type of customer. State and local governments tended to be more interested in the technology reports, most notably various volumes of the “rehab guide.” Customers affiliated with universities and research institutes and city managers tended to be more interested in program analyses. Some reports were popular with both groups, such as New Markets: The Untapped Retail Buying Power of America’s Inner Cities (U.S. Department of Housing and Urban Development, 1999c) and Now Is the Time: Places Left Behind in the New Economy (U.S. Department of Housing and Urban Development, 1999d), both published in 1999. The importance of the website has increased since 2001. It now pro- vides access to some 20 data sets, three periodicals, and about 800 publica- tions. About 60 publications are added to the website each year. (The count of data sets is potentially misleading, since a given survey is likely to include separate data for more than 1 year or more than one place. The AHS, for example, is counted only as “national data” and “metropolitan data”; there have been over 20 national surveys annually or biennially since 1973, and separate surveys for 60 metropolitan areas.) Besides listing the PD&R pub- lications, the website includes a bibliography of 10,000 items about HUD   Projects in qualified census tracts are entitled to higher tax-credit rates. 

DISSEMINATION 143 programs or urban issues. Because of the importance of the website, the committee has concentrated on evaluating its performance. The 2005 independent evaluation of the website (Bansal et al., 2005), including a survey of website users, was conducted under the Govern- ment Performance and Results Act. It found broadly similar patterns of demand to those in 2001. Data sets were the most popular, particularly income limits and fair market rents. Among the research products, tech- nology reports were the most frequently downloaded publications, with two manufactured housing construction guides and the Fair Housing Act design manual (concerning accessibility for handicapped individuals) at the top of the list; Creating Defensible Space remained popular 9 years after it was published. Research reports on homelessness and the worst case needs report were also popular. The most popular publication, by a wide margin, was the Permanent Foundations Guide for Manufactured Housing (U.S. Department of Hous- ing and Urban Development, 1996a). There were some 300,000 hits and 8,000 downloads, more than double the second most popular publica- tion in each case. This popularity was somewhat surprising, because the guide was published in 1996 but did not appear on the list of the 20 most popular publications in 2000. In addition, manufactured home production and sales were much higher in 2000 than in 2004. There appear to be two contributing explanations. Manufactured housing program staff indicate that enforcement efforts increased between the mid-1990s and early 2000, creating more demand for the Guide. In addition, the report became more accessible on HUD USER: Prior to 2003, it was available only as an execut- able file (.exe), and in that format was not frequently accessed. In 2003 it was made available in a more accessible format (as a .pdf file) on the website, and usage increased dramatically. The more important purpose of the external evaluation was to sur- vey users about the quality of the website (rather than the quality of the research). The evaluation reported very high satisfaction among users along a number of important dimensions. But 39 percent believed that the search function should be improved, and 29 percent believed that the data set search should be improved. The committee shares those concerns. There are particular difficulties in locating publications on the website. Publications are listed both alphabetically and alphabetically by topic, with a choice of 14 topics. There is also a search engine to find resources for a given subject, such as a particular HUD program. None of these resources is as helpful as it could be. Alphabetizing by title is not helpful. Publication titles do not usually   his T assessment did not cross-classify downloads by type of user, unlike the 2001 assessment. 

144 REBUILDING THE RESEARCH CAPACITY AT HUD start with words that clearly identify the subject of the publication. The first publication in the alphabetical list is A Study of Market Sector Over- lap and Mortgage Lending. It is one of nine publications whose titles start with the word “study” or “studies.” (The others are all found under “S.”) There are 11 publications starting with “assessment,” and 16 starting with “evaluation.” Basically, it is necessary to know the title of a report in order to find it. The list by topic is not much more helpful. Topics are broad and over- lapping, and topic names are somewhat idiosyncratic. The topic of “Fair Housing and Housing Finance” is not especially helpful to those who are interested in either subject. There is some overlap between “Community Development” and “Economic Development,” making it necessary to check both topics for most users. All the major housing assistance programs are in one alphabetical list, by title, under “Public and Assisted Housing.” Once within that broad category, there is no easy way to find publications on a particular program. Indeed, there is only one topic for which the reports are listed by subtopic: “Housing Production and Technology” groups publica- tions under eight headings. At the other extreme, the set of papers reporting on the GSEs’ role in funding affordable mortgages, and other topics in housing finance, can be found only under the heading “Housing Finance Working Paper Series.” The 18 publications are not listed individually. Thus a person who learns of the paper from some other source and knows only its title cannot find it on HUD USER. The other two major series of publications, the worst case needs reports and the state of the cities reports, are listed both individu- ally by title and under the general headings of “Rental Housing Assistance Reports” and “State of the Cities (National Urban Policy Report),” respec- tively, though the most recent reports are not listed. Trying to find information about a particular program through the search engine creates an opposite problem. Typing in a currently active program such as “HOPE VI” or “LIHTC” or “Section 811” results in a list of about 100 publications for each program, without much guidance about the content of a publication so that readers can make reasonable judgments as to which item is likely to be the most useful. Each of these programs has been in existence for about 15 to 20 years. Over 200 are shown for the long-established but relatively small Section 202 program. With an overall total of 800 publications covering a dozen years, it is clear that the list of 100 for each program includes many where the program is mentioned in passing. At the extreme, typing in “voucher” returns over 650 listings.   erhaps for that reason, committee members with particular interest in technology found P HUD USER easier to use than members with a primary interest in HUD programs and hous- ing or urban policy. 

DISSEMINATION 145 HUD USER contains an “advanced search” function, which is more useful but still has significant limitations. It can be used to list all publica- tions with the name of the program in the title; the resulting list usually turns out to be a subset of the major reports concerning the program. The “National Evaluation of the Shelter Plus Care Program” does not appear in the list of 18 publications with the word “homeless” in the title. There are 37 publications with the word “voucher” in the title, and 13 with the phrase “housing choice voucher,” but neither list includes any of the reports on the MTO demonstration, one of the two most recent major voucher evaluations. There are also various idiosyncratic features. LIHTC appears in the title of 12 publications, although it is necessary to type in “Low- Income Housing Tax Credit” as well as “LIHTC” to find all of them. Fre- quently, the same publication is listed separately two or more times: the full report is listed and an abstract with a link to the full report. In at least two instances, individual chapters of a report are listed as separate reports. It is not possible to search by author and not very useful to search by keyword, both highly useful and both commonly available on the websites of other bibliographic resources, such as scholarly journals. Asking by keyword turns up ten publications for “homeless” programs, six for the voucher, and no more than two for any of the other programs. It should be possible to categorize HUD’s research publications on these bases fairly quickly at a relatively small expense, if the task of categorizing by key word is assigned to someone with substantive knowledge of the relevant subject area. In general, it is much easier to find a report on the website of the con- tractor who produced it—e.g., Abt Associates, or the Urban Institute—but of course it is then necessary to know the contractor. For a given report, once located, the website typically offers two types of information: a one- or two-paragraph description of the report, with- out findings, which is reached when the report title is clicked; then, from the description, a link to the full report itself. These are both useful, but anyone interested in a brief statement of the report findings will not be well served by either. This is particularly likely to be the case for policy makers. Most reports contain an executive summary, but this is seldom published separately from the full study, making it necessary to download a lengthy document to obtain a 10-20 page summary. It would be useful to provide access to executive summaries separately from reports. Moreover, the executive summaries are often written more for scholars rather than policy makers or practitioners, and some are fairly lengthy, especially for   ddly and perhaps ironically, the 2005 evaluation of HUD USER is itself not available on O HUD USER, although the report states that it is. Possibly, it was originally available on the website and then removed for some reason. The 2001 evaluation is available. 

146 REBUILDING THE RESEARCH CAPACITY AT HUD the major evaluations. It would therefore also be useful to provide a one- or two-page summary of the findings. The 2005 evaluation separately tabulated the views of first-time and repeat users of the website. Not surprisingly, first-time users were signifi- cantly less satisfied in every dimension. Some disparity is to be expected, but the committee believes that the satisfaction of both groups can be improved, and first-time users can be encouraged to become repeat users. The HUD USER bibliography is also difficult to use. A random com- parison of a few topics, mainly programs, results in more references among the 800 PD&R publications on the website than among the 10,000 bibli- ography items. In addition, there is no search category for “author.” It is possible to search by individual, but such a search does not turn up publi- cations by an individual, only references to publications by the individual. As a convenient exercise, the committee looked at the listings for its mem- bership. For the 14 members combined, there are 28 listings. The range is from zero (for eight members) to 10. In that case, four of the listings are for comments on papers written by the individual, and the other six are listings of the authors of papers that were published in conference volumes or issues of Cityscape. The committee also looked at the listings for Richard Muth and Edwin Mills, distinguished urban economists who have both written about housing and urban policy for more than 40 years. There are three listings for Richard Muth, none for Edwin Mills. Nor is it possible to search by discipline or publication. Researchers interested in locating economic or sociological analyses of a subject or program must identify them from the full set of items turned up in the bibliography. Researchers interested in reports on a program from the U.S. Government Accountability Office must conduct the same exercise. CONCLUSIONS AND RECOMMENDATIONS Although PD&R’s information dissemination activities have much to recommend them, there is considerable room for improvement. PD&R should consider an assessment of its current efforts with the goal of devel- oping a strategically focused, aggressive communications plan. In particular, the committee believes that more can be done to identify the needs of vari- ous relevant external constituencies, package information in appropriately differentiated ways, and take advantage of new web-based technologies. The committee’s effort to review samples of recent PD&R research strongly suggests that the office needs a well-designed and continually updated management information tracking system that describes its in- house and external research. While PD&R staff ultimately were able to   ata D from Bansal et al. (2005, p. 64); statistical comparison by the committee. 

DISSEMINATION 147 respond to the committee’s multiple requests for information about PD&R research, this information was sometimes difficult or impossible to retrieve, or was incomplete. For example, the list of PD&R-funded studies since 2002 by small businesses and of evaluations conducted since 1999 included a large number of entries noting “disposition unknown.” It would be valuable if every PD&R study were entered into an easily accessible data base, and each entry accompanied by an abstract stating the purpose, methods, and findings (in addition to the information currently included such as the date and size of the award, the contractor, the govern- ment technical representative, and whether PD&R published the report). It would also be useful to know where else the study had been published and to provide a link to both the PD&R and all other publications of the study. PD&R’s dissemination formats could usefully be expanded and tar- geted to various potential audiences. As noted, HUD USER offers a choice of extremes: the full text or a very brief description that reports no results. Research Works does contain shorter and more readable summaries, but these are not available in any other format. Both of these formats could be published separately on HUD USER, making studies more readily accessible. The HUD USER website is a valuable resource to both policy makers and the public, but its current configuration has neither kept pace with technical developments in web technology nor taken full advantage of current web capabilities for data dissemination and use. The committee does not include website designers among its membership, but we believe that it should not be hard to design a more user-friendly search function, and thus a more useful website. Several members of the committee have extensive experience with websites in other fields, particularly health policy, and believe that HUD USER falls short of the norm in terms of being user friendly and calling attention to new publications and data. It appears that the PD&R website was designed some years ago and then set free to run with minor modifications; under such conditions any website becomes dated very quickly. As this report has stated in several contexts, the ­quality of PD&R’s research is generally high, and it is important to make that research accessible to interested individuals and organizations. Major Recommendation 6: PD&R should develop a strategically focused, aggressive communication plan to more effectively disseminate its data, research, and policy development products to policy makers, advocates, practitioners, and other researchers. 

148 REBUILDING THE RESEARCH CAPACITY AT HUD Recommendation 8-1: PD&R should modernize the HUD USER website. Recommendation 8-2: The HUD USER website should be made more user- friendly, enabling users to locate HUD publications and data sets more easily. It should be possible to identify publications by author and subject (including individual HUD programs) more easily. Recommendation 8-3: The bibliography available on HUD USER should allow users to search by author, discipline, and publication. Recommendation 8-4: Both internal and external research reports should be brought to the attention of interested readers more aggressively, with more accessible summaries.