|
Items in cart [1] |
Questions? Call 888-624-8373 |
PAPERBACK PDF BOOK Free Resources |
||||||||||||||||
|
||||||||||||||||
|
|
|||||||||||||||
| ||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 227
8
Management and Oversight
Many elements combine to make a cost-effective and efficient survey pro-
gram that meets the needs of its users for relevant, high-quality, timely, and
accessible information on the topics within its mandate. Some of these
elements are well-designed questionnaires; motivated respondents; capable
interviewers; talented systems staff; innovative experiments and evaluation
studies; knowledgeable analysts; and state-of-the art data collection, pro-
cessing, and dissemination technology. Not usually considered in the re-
view of a survey program is the management of the survey, including the
means for obtaining feedback and advice from others. However, an effec-
tive management structure underpins all of the above elements and is a key
component to a survey's success.
In this chapter we review the current management structure for SIPP at
the Census Bureau, including the channels through which feedback from
users and others outside the SIPP program is sought. Such feedback is
essential to keep a survey program oriented to the concerns and needs of its
users and also up to date with the latest improvements in survey design and
methods.
We believe strongly that a different and more effective management
structure is needed if SIPP is to achieve its full potential in the future. In
our view, the Census Bureau should not treat SIPP as "just like any other
Census Bureau survey" from a management perspective. SIPP is far more
complex than the Bureau's other household surveys. More important, the
Census Bureau has leadership, analysis, and dissemination responsibilities
227
OCR for page 228
228
THE SURVEY OF INCOME AND PROGRAM PARTICIPATION
for SIPP above and beyond its responsibilities for collecting and processing
the data. In this regard, the Census Bureau's role for SIPP resembles its
role for the decennial census and differs from its role for other household
surveys: the Bureau both sponsors and operates the decennial census but
typically conducts household surveys on behalf of an outside sponsor agency.
The Census Bureau must manage SIPP in a manner that is commensurate
with its responsibilities as the survey's sponsor and as the lead federal
agency for analysis and publication of statistics on family and individual
income. (The latter role also encompasses management of the March Cur-
rent Population Survey (CPS) income supplement.)
MANAGEMENT STRUCTURE
Description
The responsibility for household surveys that are conducted by the Census
Bureau (including SIPP, CPS, the Consumer Expenditure Survey, and oth-
ers) is lodged with Demographic Programs~ne of six major organiza-
tional units within the Bureau, each of which is headed by an associate
director. (See Figure 8-1; subsequently, for clarity, we refer to Demo-
graphic Programs as the "Demographic Directorate.") The Demographic
Directorate in turn is organized into six divisions, two divisions that deal
with international programs and four that are involved with U.S. household
surveys: Demographic Surveys Division (DSD), Housing and Household
Economic Statistics (HHES), Population (POP), and Demographic Statisti-
cal Methods Division (DSMD).
DSD performs many functions for each survey, including general man-
agement and coordination, questionnaire design, and data processing. Staff
of other divisions both inside and outside the Demographic Directorate-
also play important roles. Within the Demographic Directorate, DSMD
staff are responsible for sample design and selection and most methodologi-
cal research and evaluation, while HHES and POP staff are responsible for
publications and liaison with users. Outside the Demographic Directorate,
the Field Division manages the interviewing staff, while the Center for
Survey Methods Research and Statistical Research Division contribute to
methodological research. Data User Services Division (DUSD) fills orders
for public-use microdata products and documentation (primarily for SIPP
and the March CPS, among the Bureau's household surveys). DUSD will
also develop user products, such as users' guides, but only by request and
with funding from another division.
1HHES and POP perform these functions for the March CPS as well as SIPP, but not
generally for surveys that are conducted for other federal agencies.
OCR for page 229
229
-
4' E E0
C) °C ~
.m ~ ° I
O ~ ~
~'
l
U) .o
~ .m
dO
E 2
o
W "C
_ I _ _ I I
~-F~m ~ ;] '
-
~ a,
.O
as ~
C `i' o
-5
·_
as
CO
o
_ :>
~ss C)
~ E
Cd
c5
, ~.
. 1 1
_
. ~
o
~ o-E
o
~ Q
e
1 _
giF_ 97
o
·-C-~
(D O.
~o
._
0 .a,
o _
0328 ~
1
3-S ~
e 1:
l
I D
1 'L
_
J~
I cO
,
1 0 o
!~0-m
1
1 ~
1 ~ ~
I o c"
1 E~C
1 ~ '
I lS 0 ~
1 o c
1 o,
1 0
1 ~
1
-
s~
Ct
v
o
.
-
C~
.
o
:'
C~
C)
~s
m
C~
C~
v
~0
8
a5 C
~d
~C:
.l
OCR for page 230
230
THE SURVEY OF INCOME AND PROGMM PARTICIPATION
The responsibility for coordinating the work and generally for "making
the trains run" for each household survey is typically lodged with a branch
chief in DSD. One of the branch chief's functions is to obtain budget
requests from all divisions with involvement in the survey and funnel them
to the head of DSD and, ultimately, the head of the Demographic Director-
ate, for review and adjudication.
Because SIPP involves staff in so many divisions in the Census Bu-
reau both inside and outside the Demographic Directorate and because
of its relative newness and complexity, several cross-cutting committee structures
have been tried to improve communication and coordination of-the work
Initially, the income surveys (SIPP) branch chief in DSD chaired a commit-
tee of representatives from each division that met frequently to keep every-
one up to date. Also, the chief of POP (HHES did not yet exist) was
delegated wide authority to monitor and guide the survey. Although not in
charge of the survey's budget, he was instrumental in ensuring that a por-
tion of the funding was available for methodological research and experi-
mentation. He also played a deciding role in several key changes to the
questionnaire (e.g., the decision to combine several scattered topical mod-
ules into a single personal history module administered early in each panel).
As problems developed (principally with data production and user liai-
son), it became clear that a stronger coordinating mechanism was needed.
Staff committees were set up to address specific areas, such as user needs
and methodological research. An executive committee, consisting of the
chiefs of divisions with a major role in SIPP, was charged to review propos-
als from the staff committees and make major decisions for the survey.
(The executive committee was chaired at times by the head of the Demo-
graphic Directorate and at times by a special assistant to the head.)
Some of the staff involved with SIPP work solely on the SIPP pro-
gram-for example, the staff in the DSD income surveys branch and the
DSMD SIPP branch.2 However, SIPP analysts in HHES and POP typically
have other responsibilities as well-for example, analyzing data from the
decennial census and supplements to the CPS. Initially, POP included sev-
eral staff with a strong focus on SIPP who also served as contact points for
questions and problems from outside users and who initiated research and
development projects for SIPP. Subsequently, some of these staff were
attached to the associate director's office. Plans were made to put these and
other staff in an expanded Center for Demographic Studies within the De-
mographic Directorate that would conduct analyses of longitudinal data from
SIPP and other surveys, but those plans were dropped, and the Center was
disbanded. At about the same time, several staff with a long association
2The DSD income surveys branch, despite its title, deals exclusively with SIPP and not also
with the March CPS income supplement.
OCR for page 231
MANAGEMENT AND OVERSIGHT
231
with SIPP left the Census Bureau or the SIPP program, and some of them
were not replaced. Recently, a SIPP user liaison position was established in
HHES, which is the division that analyzes the core income and program
participation data from SIPP. However, no additional positions for user
liaison or SIPP analysis have yet been allocated.
Assessment
The management structure for SIPP, as for other household surveys, follows
well-established practice at the Census Bureau. Given the Bureau's long
record of solid performance in many aspects of survey operations, we do
not lightly offer criticisms. However, we believe that the structure does not
serve SIPP well.
The basic decentralized staff organization for SIPP and other Census
Bureau surveys follows a widely accepted model. Many agencies and firms
that conduct surveys organize their staffs by function for example, with
the field staff in one division, the data processing staff in another, the
survey and sample design people in another, and so on. This type of struc-
ture facilitates such important functions as staff training, mentonng, and
career development. However, problems arise in taking the staffs from
different divisions who are to participate in a particular survey project and
turning them into a cohesive, efficient, well-focused team. Moreover, the
work of the team must be driven first and foremost by the survey's goals,
which are ultimately determined by the analysis and research needs of the
users.3
Most survey organizations address this challenge by naming a project
director or principal investigator for the survey who is a senior person with
relevant substantive background and survey experience. (A variant struc-
ture is to have a subject-matter-oriented principal investigator paired with
an operations-oriented survey director.) This individual is assigned overall
budget and management authority for the project and is designated the leader
of the team. Most other staff members on the team retain a "home base" in
their own division but are responsible to the principal investigator for their
work on the survey. Often, members of one team will also participate in
teams for other survey projects. However, if a particular survey is big and
complicated enough, there will typically be a core staff who work full time
(or close to full time) on it. This core staff will include task leaders (e.g., a
leader for data processing or survey design). It will also almost always
include analysis staff who play a major role throughout the project-in
3See Scott (1987:Ch. 9) for a discussion of the challenges that organizations face in devel-
oping an appropriate division of labor for particular kinds of technical work and at the same
time coordinating the efforts of multiple staff across divisions in order to carry out a project.
OCR for page 232
232
THE SURVEY OF INCOME AND PROGRAM PARTICIPATION
specifying questionnaire content and edits and imputations in addition to
analyzing and reporting the survey results.
In contrast, the Census Bureau assigns general management responsi-
bilities for its surveys to DSD staff at the level of a branch chief, which is
several steps down in the hierarchy. As needed, committees are set up to
facilitate communication and coordination of work across divisions. How-
ever, there is no equivalent of a principal investigator leading a designated
team, and staff in different divisions often work at arm's length from each
other (particularly analysts and data processing and other survey staff).
This mode of operation can work well for the household surveys that
the Census Bureau conducts on behalf of an outside sponsor agency, since
the sponsor agency provides the substantive focus and an impetus for coor-
dination. Typically, that sponsor agency whether it is the Bureau of Labor
Statistics (BLS) for the Consumer Expenditure Survey and the main labor
force portion of the CPS, the National Center for Health Statistics (NCHS)
for the Health Interview Survey, or the National Center for Education Sta-
tistics (NCES) for the Schools and Staffing Survey sets the survey goals
and content, exercises authority over the budget, has substantial input on
design issues, analyzes and evaluates the data, prepares publications and
microdata products, and provides support services and liaison to users.
In the case of SIPP, the Census Bureau itself is the sponsor agency, with
full budget authority and responsibility not only for survey operations, but
also for the content, analysis, and user service functions. We believe that
these added areas of responsibility necessitate strengthening and focusing
the management of the program. Evidence for our assessment comes from
examining the start-up problems that SIPP experienced, which were most
severe precisely in those areas of responsibility: for example, the lag in
developing a publication program for the core data on income and programs
and the setbacks experienced in developing adequate microdata products,
documentation, and other services for users. We recognize that many fac-
tors hindered SIPP at the outset, not least the externally imposed significant
budget cuts. Nonetheless, we believe that a stronger, more focused man-
agement structure would have made it easier for the Bureau to operate so
complex a survey and to respond more quickly and effectively to problems.
Looking to the future, the Census Bureau faces a major management
challenge in planning for and implementing the redesign of SIPP in a timely
and cost-effective manner. The redesign will affect all aspects of the sur-
vey from content and design to collection and processing to analysis and
dissemination. Moreover, the complexity and scope of the survey will con-
tinue to make it challenging to manage even after the redesign is in place.
Finally, it is critical for the survey to restore its capacity for flexible re-
sponse to changing social welfare policy and research data needs, but a
decentralized structure makes it difficult to accommodate flexibility and
OCR for page 233
MANAGEMENT AND OVERSIGHT
233
change and at the same time achieve operational efficiency. For all these
reasons, we believe that it is imperative for the Census Bureau to change its
management approach to SIPP.
As noted above, the Census Bureau has experimented with more struc-
tured coordination and decision-making arrangements for SIPP, indicating
that it has not been completely comfortable with decentralized management
of the survey. Currently, there is a SIPP executive committee comprised of
relevant division chiefs and chaired by the head of the Demographic Direc-
torate, with staff subcommittees on particular topics. An assistant division
chief in DSD provides staff support to the executive committee. - However,
this structure is not the same as a dedicated SIPP team. Moreover, it is
awkward to have a Bureau associate director who is acting, in effect, as the
principal investigator or project director for SIPP, given the general man-
agement responsibilities of this position for all of the Census Bureau's de-
mographic surveys and programs.4
Recommendation: A Different Approach
The panel considered alternative management arrangements for SIPP. One
model would be to lodge the sponsorship with another agency and so malice
SIPP like the other household surveys conducted by the Census Bureau.
The original Income Survey Development Program (ISDP) was a joint ven-
ture of the Office of the Assistant Secretary for Planning and Evaluation
(ASPE) in the Department of Health and Human Services (DHHS) and the
Census Bureau, and the original plan for SIPP was to have the Social Secu-
rity Administration (SSA) settee as the sponsor agency and the Census Bu-
reau as the operating agency.
However, it was the Census Bureau that came to the rescue of SIPP in
the early 1980s, and it was then assigned full responsibility and budget
authority for the program. To its credit, the Bureau successfully launched
the survey, which, despite its problems, has provided invaluable data for
critically important policy and research studies. Furthermore, the Census
Bureau has a long history as the lead federal agency for analysis and publi-
cation of statistics on family and individual income-one of the two princi-
pal topic areas of SIPP. These are excellent reasons to keep SIPP at the
Census Bureau.
Moreover, there is no other obvious candidate to serve as the sponsor
agency. Responsibility for federal assistance programs-the other main
topic area of SIPP is divided among several operating and policy analysis
4Recently, the associate director delegated some planning and decision authority for the
redesign to a committee comprising the division chiefs of DSD, HHES, POP, and DSMD. This
group is meeting weekly to consider redesign issues and milestones.
OCR for page 234
234
THE SURVEY OF INCOME AND PROGRAM PARTICIPATION
agencies (ASPE, SSA, and the Administration for Children and Families in
DHHS and the Food and Nutrition Service in the Department of Agncul-
ture). None of these agencies is as well positioned as the Census Bureau to
take a broad, objective view of the survey or to resist pressures for politi-
cizing the program, and none of them is strong in survey methodology.
Other statistical agencies, such as BLS, have limited experience in the topic
areas covered by SIPP.
Given the strong arguments to continue the Census Bureau as both
sponsor and operator of SIPP, the question is how to create an internal
group that cart provide the needed substantive focus and weld the survey
staff into a cohesive, efficient, and committed team. There are undoubtedly
several organizational arrangements that would serve this end, and the Cen-
sus Bureau will have to weigh many factors in developing an organization
scheme and operating plan to achieve the goal of a stronger and more
focused management for SIPP.5 However, there He several points that we
believe are key to the success of the overall concept. Most important is the
need for a project director with full management and budgetary authority
for the Bureau's income surveys (SIPP and the March CPS income supple-
ment). We include the March CPS because it has been the mainstay of the
Census Bureau's income statistics program in the past and will continue to
make an important contribution for some years to come.6
To be in the best position to serve as an effective sponsor for SIPP, the
project director should be at a very senior level: most likely, within the
Census Bureau's current organizational structure, an associate director or
another position that reports directly to the deputy director. The project
director should be someone who combines considerable substantive policy
analysis or research experience on such topics as the distribution of income,
poverty, and the dynamics of program participation with a strong back-
ground in management of complex projects involving original data collec-
tion. Finally, the project director needs to have sufficient resources for an
analysis staff that is large enough to fulfill the Census Bureau's leadership
50ne possibility would be for the Bureau to set up the equivalent of a "National Center for
Income Statistics" within the Census Bureau, headed by a senior person reporting to the deputy
director. The center could include the income and program analysis staff, as well as team
leaders for the various staff groups (e.g., demographic surveys, field, statistical methods, etc).
6The March CPS income supplement (along with some demographic supplements to the
CPS) is the only other household survey for which the Census Bureau serves as the sponsor as
well as operating agency. However, the arrangement is not quite the same because the March
supplement piggybacks on the main CPS, which is sponsored by BLS. Also, the March
supplement is a much simpler data set than SIPP. Nonetheless, we believe that combining the
overall responsibility for the March supplement and SIPP under the some project director will
benefit both surveys and, more generally, the Bureau's income statistics program.
OCR for page 235
MANAGEMENT AND OVERSIGHT
235
and dissemination responsibilities for income and program statistics from
SIPP (and the March CPS).
We see important advantages from having a focused management struc-
ture for SIPP, headed by a project director who is oriented to income and
program statistics issues and who has the necessary budget and management
authority for the Bureau's income surveys. Such a structure would provide
a strong source of substantive guidance and direction for SIPP and the
March CPS. It would also facilitate full involvement of the analysis staff
with the survey staff in many aspects of survey design and operations (e.g.,
questionnaire content and editing decisions). It would better position the
staff both analysts and survey people to communicate on an equal foot-
ing with analysts in other agencies and with academic researchers about
income and program concepts, data needs, measurement methods, and analysis
techniques. Also, it would foster clear and effective channels for users to
communicate problems, quenes, and suggestions and obtain timely feed-
back. Finally, such a focused structure would facilitate the development of
in-service training and other means for the staff to keep abreast of the latest
conceptual and methodological developments related to income and pro-
gram statistics-whether new methods to exploit the richness of SIPP long~-
tudinal data for spell analysis or new techniques for improved measurement
of asset income sources. In all of these ways, the new structure should
enable SIPP to operate in a cost-effective manner and at the same time
adapt in a timely manner to changing policy concerns and data needs.
We note that there are precedents within the Census Bureau for the type
of dedicated management structure that we are proposing for SIPP. In the
mid-1980s, a new associate director position was created to manage the
1990 decennial census, and a number of divisions were moved into this
office. This step made sense, given that the census is not only the Bureau's
largest operation, but also one for which it serves as the sponsor as well as
the operating agency.
Recommendation 8-1: To be as effective as possible in carrying
out its responsibilities to produce timely, comprehensive, rel-
evant, high-quality, and analytically appropriate statistics on
income and program participation, the Census Bureau should
establish a senior-level position of project director for the Bureau's
income surveys, SIPP and the March CPS income supplement.
That position should include full management and budgetary
authority for the income statistics program and sufficient re-
sources to obtain the level of analysis staff that is needed to
provide substantive guidance to the program, prepare reports,
conduct analyses, and evaluate analytical concepts and meth-
ods. The person who fills this position should have recognized
OCR for page 236
236
THE SURVEY OF INCOME AND PROGRAM PARTICIPATION
substantive expertise in topics related to income, poverty, and
assistance programs, combined with strong survey management
skills.
OVERSIGHT
Throughout SIPP's history, the Census Bureau has shown commendable
initiative in seeking input from data users and survey experts on various
aspects of the SIPP program. An array of advisory groups and input mecha-
nisms has existed at one or another time, and these groups have made
valuable contributions to the survey. Indeed, there is much in the SIPP
advisory process that can serve as a model for other surveys. There are also
ways in which the advisory mechanisms for SIPP can and should be im
proved.
We briefly describe past and present advisory mechanisms for SIPP and
suggest ways to improve the oversight function in the future. We cannot
stress enough the importance of obtaining and malting the best use of out-
side advice and feedback to keep a complex operation such as SIPP on
target and able to meet challenges and opportunities. An advantage of the
focused management approach that we recommend is that it should facili-
tate the development of an improved advisory function for the Census Bureau's
income statistics program.
Outside Input: Past Experience
The Census Bureau has sought advice on SIPP from federal agency users,
academic researchers, survey methodologists, and experts in data access
and use of microdata products.7
Federal Agency Users
The Census Bureau has used several mechanisms to obtain input on the
content and other aspects of SIPP from the perspective of federal agencies
that use SIPP data for policy analysis and research. A principal vehicle for
input has been an interagency committee, chaired by the Statistical Policy
Office of the U.S. Office of Management and Budget (OMB). The Census
Bureau has also obtained agency input through requests to agency staff for
written comments and discussions with the D.C. Users' Group, an informal
The panel's description and assessment of advisory mechanisms for SIPP, particularly the
interagency committee chaired by the Office of Management and Budget, draws on the interim
report on SIPP of the Committee on National Statistics (1989). In addition, panel members
and staff have participated in meetings and other activities of the various SIPP advisory groups.
OCR for page 237
MANAGEMENT AND OVERSIGHT
237
group of SIPP users from executive and legislative branch agencies and
contract research organizations in the Washington area who meet monthly
to exchange views and hear reports of research using SIPP data.
The OMB interagency SIPP committee was established in 1983 and
assisted in developing and revising the SIPP questionnaire, including the
core and topical modules. Among other aspects of SIPP reviewed by that
committee (or a series of its subcommittees) were file structures, longitudi-
nal household definitions, self- versus proxy response, nonmetropolitan iden-
tification, plans for reducing the sample size, and comparability with the
CPS.
At present, the OMB SIPP committee has no regular meeting schedule.
It has experienced long periods of inactivity for example, the committee
did not meet from July 1986 until June 1989. This situation was due in part
to constrained resources and other problems affecting the statistical coordi-
nation function within OMB and in part to a narrow perspective on the part
of the Census Bureau of the committee's role. The Bureau has viewed the
OMB committee as having a primary role in specifying the content of the
variable topical modules in SIPP that are designed to respond to agency
needs for data on emerging policy concerns, but as having a more advisory
role with regard to the content of the core and fixed topical modules (e.g.,
on assets) or other features of SIPP.
Academic Researchers
The Social Science Research Council (SSRC) provided funding from 1982
through 1988 for a committee on SIPP that included members of the aca-
demic research community with an interest in longitudinal data for social
welfare policy analysis and research. The SSRC committee was initially set
up following a conference in fall 1982 on the technical, conceptual, and
administrative lessons of the ISDP, sponsored by the SSRC Center for Co-
ordination of Research on Social Indicators (David, 1983~.
The SSRC committee hosted several symposia (e.g., see David, 1985b)
in which contributors addressed ways in which SIPP could be enhanced to
better serve academic research needs in a variety of topic areas, including
family structure, income distribution, labor markets, education, women and
children, minorities. The committee later obtained Census Bureau funding
for a conference on individuals and families in transition, which featured
papers using SIPP and other panel surveys for longitudinal analysis (Bureau
of the Census, 1988a). The committee met two or three times a year with
Census Bureau staff in 1983-1988 to discuss various aspects of the SIPP
program.
The SSRC committee clearly played a very important role in making
the research community aware of SIPP and encouraging use of the data (see
OCR for page 238
238
THE SURVEY OF INCOME AND PROGRAM PARTICIPATION
Appendix A). The committee and its symposia and conferences were also
fertile sources of ideas for improvements to the SIPP design and content.
However, the committee's relationship with the Census Bureau was not
always smooth. The Bureau necessarily had an operational perspective on
SIPP and a primary concern with running the survey and meeting deadlines.
Suggestions from academics about content or design changes were often
viewed as impractical, needlessly complicating an already complex opera-
tion, or outside the survey's original mandate. The Census Bureau's deci-
sion in late 1987 to freeze the core questionnaire brought into question the
contribution that the SSRC committee could make. Having achieved many
useful goals, the SSRC disbanded the committee in 1988. Since then, there
have been no formalized channels for input to SIPP from the academic
research community.
Survey Methodologists
In the first years of SIPP, the survey was often featured at the semiannual
sessions of the Census Bureau advisory committees of the American Statis-
tical Association (ASA) and American Economic Association (AEA). How-
ever, the committee members found it difficult to provide useful commen-
tary on complex methodological issues for SIPP because their time was so
limited and their charge covered so many other Census Bureau programs.
In spring 1986 the ASA committee recommended that the Census Bureau
sponsor an advisory group of methodologists solely for SIPP. (The SSRC
SIPP committee concurred and supported this development.) The Survey
Research Methods (SRM) Section of ASA subsequently established the ASA/
SRM Working Group on Technical Aspects of SIPP, which held its first
meeting in fall 1986.
The ASA/SRM working group continues to meet about twice a year to
discuss such issues as maximum telephone versus personal interviewing,
longitudinal weighting concepts and techniques, and the results of record-
check studies and other evaluations of the quality of SIPP data. As noted in
Chapter 7, the group was a major instigator of the SIPP Quality Profile.
The group has developed good working relationships with Census Bureau
staff and has helped guide priorities for the SIPP methodological research
and evaluation program.
Experts in Data Access and Use of Microdata Products
Problems with data files and documentation surfaced early on in the SIPP
program. For several years, the Census Bureau obtained user input on
improvements to the microdata products through informal means, for ex-
ample, consulting with the D.C. Users' Group and listening to individual
OCR for page 239
MANAGEMENT AND OVERSIGHT
239
SIPP users who had complaints and suggestions. The Association of Public
Data Users (APDU) became aware of SIPP users' problems and proposed a
joint user-Census committee to address issues involving the design and
content of SIPP microdata products, documentation, access mechanisms (such
as an on-line extract capability), and means to improve communication be-
tween users and Census Bureau staff.
The APDU committee, which includes data librarians, microdata users,
and systems analysts, along with key Census Bureau data processing and
user services staff, first met in January 1989 and has since met once or
twice a year. The APDU effort also involves publication of a supplement to
the APDU newsletter, mailed to everyone on the Census Bureau's list of
inquirers about SIPP, that provides minutes of the APDU committee meet-
ings, notices of documentation and data file changes, and other articles of
interest to SIPP users.
The APDU committee helped specify a more accessible person-month
format for the SIPP data files, which was recently implemented for wave
files from the 1990 and later panels. The committee has also made numer-
ous suggestions related to documentation, conventions for coding missing
data, and similar matters. Only limited progress, however, has been made
in responding to many of these suggestions, such as providing descriptions
of edits and imputations in the documentation. And in spite of the committee's
recommendation to increase the resources devoted to user liaison activities,
there are fewer staff addressing this area now than 2 years ago, when sev-
eral staff members who worked most closely with users left the SIPP pro-
gram.8
Other Sources of Advice
Formal advisory groups have not been (and should not be) the only source
of input to SIPP. Other channels for Census Bureau staff to interact with
others, encourage use of the data, and obtain feedback and ideas (see Chap-
ter 6) have included: sessions at professional association meetings that
featured SIPP-based research or methodological work; the ASA/Census re-
search fellowship program, which attracted a number of researchers who
used SIPP data on-site at the Census Bureau and provided their views on
the program; and presentations and training sessions about SIPP given by
Census Bureau staff (e.g., as part of the summer program of the Inter-
university Consortium for Political and Social Research). Publication se
8A SIPP Data Products Process Action Team, consisting of staff from several Census Bu-
reau divisions, was recently established to develop priorities for improvements in data prod-
ucts, documentation, and other user services that could be funded with the expected savings
from the recent switch to maximum telephone interviewing in SIPP.
OCR for page 240
240
THE SURVEY OF INCOME AND PROGRAM PARTICIPATIONr
ries, including volumes of SIPP-based papers presented at ASA meetings
and the SIPP Working Paper series, have also served to build the SIPP user
community and encourage interchange among users and Census Bureau staff.
Indeed, since SIPP was initiated, the Bureau staff have shown exceptional
initiative in developing these kinds of communication channels. However,
users have expressed concern that there are not sufficient resources at the
Census Bureau to further these activities now that there is only a single
SIPP liaison position in HHES.
Observations and Directions for the Future
We have a number of general observations about the experience with out-
side advice for SIPP. First, it appears that the advisory process that has
worked best is the ASAISRM working group. The Census Bureau has a
long tradition of seeking input on technical issues related to survey research
and evaluation, and the staff are comfortable in working with outside ex-
perts in such areas as questionnaire design, sampling, weighting, and impu-
tation. The ASA/SRM group is largely made up of survey methodologists
who are not themselves data users and hence have no particular interest in
steering SIPP in any particular direction The group's members and the
Census Bureau methodological staff have been able to develop a collegial
working relationship built around a common goal of finding ways to im-
prove the quality of the SIPP data.
Other SIPP advisory groups have had somewhat less success in devel-
oping strong working relationships with the Census Bureau. Different per-
spectives-for example, the viewpoints of outside researchers or policy analysts
and Census Bureau operations people have been a problem in some in-
stances. Limited progress in such areas as documentation (whether due to
resource constraints or other factors) has been another source of tensions.
Finally, there are some gaps in the advisory structure for SIPP. Cur-
rently, there are no formal channels for input from academic researchers,
who are important users of SIPP and important resources for keeping SIPP
abreast of the state of the art in many areas. Also, there has never been an
advisory group on technical issues relating to the analysis of SIPP cross-
sectional and longitudinal data-for example, such issues as the definition
and measurement of spells of poverty or program participation or appropri-
ate ways to broaden the definition of income. (The ASAJSRM working
group has considered weighting and imputation strategies but not other esti-
mation issues.)
We commend the Census Bureau's initiatives in seeking input on SIPP
from a broad range of perspectives, but we conclude that the Bureau could
usefully strengthen and clarify the advisory process in a number of respects.
We believe that success in this regard depends, first, on adopting a focused
OCR for page 241
MANAGEMENT AND OVERSIGHT
241
management structure for the program as a whole. Such a structure would
foster improved internal communications and cohesion among the various
staff groups that work on SIPP, which, in turn, would make it easier to
provide timely feedback to users. Also, strengthening the analytical aspects
of the Census Bureau's income statistics program would put the staff in a
better position to interact productively with outside researchers about con-
tent issues and technical matters relating to data analysis and presentation.
With regard to specific improvements to the SIPP advisory function, we
emphasize that there is no single best way to organize the input process and
that informal as well as formal mechanisms have a role. In particular, we
are concerned that the Census Bureau not assume that the only way to fill
gaps is to create additional expert review groups. We urge the Bureau to
think creatively about the advice mechanisms and structures that are likely
to be most cost-effective, particularly in the context of a new management
approach for SIPP.
Broadly speaking, the Census Bureau needs to obtain two kinds of
input for SIPP on a continuing basis: advice from policy analysts, research-
ers, and other users on issues of goals, direction, overall design, and con-
tent; and advice from technical experts on matters related to survey method-
ology, analysis, and data products and dissemination. In addition, as noted
in other chapters, the Bureau needs mechanisms to ensure that all of the
staff analysts, survey methodologists, and operations and data processing
people have opportunities for regular in-service training to update their
skills and knowledge as well as to learn from outside peers. Finally, as we
note in Chapter 6, the Bureau needs to continue to encourage the use of
SIPP data through a variety of means and to improve its communications
with individual users about their problems and successes in working with
the data products.
Advice on Goals, Content, and Basic Design
Federal agencies concerned with social welfare policy are obviously key
constituents of the SIPP program, and the Census Bureau must regularly
consult their views. Hence, it is important that the OMB interagency com-
mittee continue to function. The Census Bureau should work actively with
the OMB staff to put the operations of the committee on a sounder basis by
planning its agenda for a 1- or 2-year period and scheduling meetings at
least twice a year (more often, as needed). The Census Bureau should
consult the committee, if this has not already been done, about key aspects
of the proposed survey redesign.
The Census Bureau should also consult regularly with academic re-
searchers and other users outside the federal government about SIPP be-
cause SIPP is important to the development of new knowledge in the social
OCR for page 242
242
THE SURVEY OF INCOME AND PROGRAM PARTICIPATION
sciences (which is ultimately the basis for advances in policy analysis and
information for decision-making) and because federal agencies frequently
commission researchers to carry out SIPP-based policy studies on their be-
half. It could be very useful to organize an advisory group of nonfederal
agency users, on the model of the SSRC committee, that could coordinate
input from researchers about priorities for topical modules and provide other
guidance about the content and direction of SIPP. It could also be useful, as
an added or alternative means of obtaining input, to organize periodic con-
ferences to receive advice and feedback; such conferences could also fea-
ture research uses of SIPP.9
Whatever the means chosen, we urge the Census Bureau to be very
clear about the type and extent of advice that is being sought (and to pro-
vide sufficient time for thoughtful input): for example, advice on content
changes for a major redesign can be more expansive than advice on content
changes over the next couple of years. Also, we urge the Bureau to main-
tain a process to inform the people who provide input about the fate of their
suggestions. Such steps can help bridge the different perspectives and de-
velop more productive working relationships between researchers and Bu-
reau staff.
Advice on Technical Issues
The ASA/SRM working group has functioned effectively to date in provid-
ing outside review of the Census Bureau's research and evaluation program
related to survey methods and data quality in SIPP. It seems very useful for
the group to continue.
Building on this model, we recommend that a parallel working group be
established to advise the SIPP analysis staff. For this area, we believe that
an additional group is needed because of the difficult issues that the Census
Bureau must wrestle with in developing appropriate income statistics from
the complex SIPP longitudinal data. We see no reason that an analytically
focused group would not function in a similarly collegial manner as the
survey-methods-focused ASAISRM group. Such an analytical group might
be sponsored jointly by the AEA and the ASA and include economists,
statisticians, and survey analysts with conceptual and analytical expertise
related to income statistics. As part of its activities, the group could peri-
odically review the analytical content of the Census Bureau's income statis-
tics publications, including those in the research series.
Finally, we believe it is important to continue to haste some type of
mechanism, such as the APDU SIPP committee, for obtaining regular ad
9The Conference on the Future of SIPP sponsored by our study panel might serve as a
useful model; see Chapter 2.
OCR for page 243
MANAGEMENT AND OVERSIGHT
243
vice and feedback on the Census Bureau's microdata products and data
access systems for SIPP. We urge the Census Bureau to recognize, how-
ever, that more resources and coordinated staff attention will be needed to
respond to the issues that such an advisory group will raise. We urge the
Census Bureau, in rethinking its management structure for SIPP, to place
more emphasis on this important area.
Recommendation
Recommendation 8-2: We support the Census Bureau's efforts
to obtain outside advice about the SIPP program and encourage
the Bureau to further strengthen its advisory mechanisms. The
Bureau should regularly seek advice about the content, overall
design, and goals of SIPP from federal agency users and from
other users, including academic researchers. The Bureau should
also regularly seek advice about technical matters from experts
in the field. Working groups should be formed or continued in
three main areas: (1) survey methods and evaluation of ways to
improve data quality; (2) conceptual and analytical issues in the
development of appropriate income and program statistics from
complex longitudinal data; and (3) microdata products, docu-
mentation, and means of data access.
As a concluding note, we have been impressed throughout our evalua-
tion with the careful thought and attention that everyone we consulted has
given to the question of the future of SIPP. Clearly, the many policy ana-
lysts, researchers, and survey methodologists who have been involved with
SIPP, as well as the SIPP staff at the Census Bureau, support the program
and are anxious to see it improve. We urge the Census Bureau to change its
approach to the survey in ways that will take full advantage of the interest
and commitment of the SIPP community and enable SIPP to fully realize its
promise to improve the nation's statistics on income and program participa-
tion.
OCR for page 244
Representative terms from entire chapter:
project director
