Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 309
Intelligence Analysis: Behavioral and Social Scientific Foundations 13 Implementing Change: Organizational Challenges Amy Zegart Improving organizational performance is never easy. As sociologist Jim March has noted, success requires that organizations balance exploration—the search for new ways of doing things—with exploitation, the ability to harness new practices and jettison older, less effective ones (March, 1991). These challenges confront all organizations, but two factors make them more acute for intelligence agencies. The first is bounded rationality (Simon, 1976). In the theoretical world, individuals have the luxury of perfect rationality, seeing all of the relevant options, assessing trade-offs with clarity, and making the best decisions. The real world is not as nice. There, rationality is inherently limited or bounded by uncertainty, imperfect information, and cognitive constraints that lead individuals to make decisions that appear to be “good enough”—but may turn out to be nowhere close (Simon, 1976). Intelligence officials have the toughest time of all, confronting bounded rationality problems in spades. Their job is to give policy-making customers decision advantage amidst swirling uncertainty, missing information, enemy deception and denial, and fast-changing events that are often unforeseeable, even to the participants themselves. The second acute intelligence challenge is secrecy. As I discuss below, the more specialized any organization becomes, the harder it is for any one part of the organization to understand or improve what another part is doing, a phenomenon that sociologists call “structural secrecy” (Vaughan, 1996). In the classified universe, of course, this structural secrecy is compounded by actual secrecy, which protects vital information from adversaries, but also compartmentalizes information, ideas, organizations, and practices to a much greater extent.
OCR for page 310
Intelligence Analysis: Behavioral and Social Scientific Foundations Despite the intelligence community’s (IC’s) unique challenges, the fields of organization theory and political science offer useful insights and cautionary warnings about the organizational side of improving intelligence analysis. The chapters in Part II (Analytic Methods) of this volume mine an array of relevant literature for the best analytic tools to improve intelligence analysis. Here, we turn to a different task: Examining a broad sweep of relevant social science research with an eye to identifying which organizational factors impede or facilitate effective analysis. Worth underscoring, though, is the fact that social science does not offer ready-made instructions about how to make intelligence analytic improvements stick. However, it does offer some useful generalizations that can illuminate the trade-offs and challenges involved to guide more effective implementation. INSIGHTS AND LIMITATIONS OF ORGANIZATION THEORY Organization theory is a wide-ranging, multidisciplinary field that includes sociology, psychology, political science, economics, and professional school fields such as urban planning and management. Although organization theorists tackle vastly different questions using a multitude of methodologies, they all share an interest in understanding how organizations behave, and why. In general, the field’s research is animated by three central issues: (1) how internal organizational structures and features affect organizational outcomes (particularly efficiency and survival); (2) how external factors influence what goes on inside an organization; and (3) how the interaction between internal and external forces shapes an organization’s prospects for survival. For our purposes, the field offers three insights for improving intelligence analysis, described in the following pages. Insight #1: Adopting New Practices Is Difficult Even for Firms This idea is more important than it sounds. Critics frequently bemoan that government is not run more like a business, and recommend exporting private-sector practices into public-sector bureaucracies (Osborne and Gaebler, 1993; Osborne and Plastrik, 1998). The data show, however, that most businesses are not run like businesses. Consider survival, which is the most rudimentary indicator of firm adaptation (Aldrich, 1999).1 According to the U.S. Census Bureau, nearly a third of the 5.5 million American businesses that existed in 1990 failed within four years (Aldrich, 1999). 1 As Aldrich points out, such findings most likely understate adaptation failure because they focus only on surviving populations, excluding all of the organizations that never made it past the start-up phase, when survival rates are considerably lower.
OCR for page 311
Intelligence Analysis: Behavioral and Social Scientific Foundations Every year, more than half a million American businesses go bust. That’s about 1,500 per day or 1 business every minute (U.S. Census Bureau, 2009, Table 739).2 What’s more, social science research suggests that corporate fads often flop. Pfeffer and Sutton (2006), for example, note that studies repeatedly find that the majority of corporate mergers (some estimates are 70 percent or more) fail to deliver promised benefits and actually end up destroying value. Analysis of 93 studies covering more than 200,000 mergers published in peer-reviewed journals found that on average, the negative effects of a merger on shareholder value appeared within days after the merger was announced (Pfeffer and Sutton, 2006). Even top-performing firms struggle to sustain their performance. Between 1955 and 2005, for example, nearly 2,000 companies made Fortune magazine’s list of the largest 500 U.S. corporations. Of these, only three held the number one spot for more than a single year; 27 made the list once without ever appearing again; and just 71, or 3.8 percent, managed to stay on the list for the entire 50-year span (Schlosser and Florian, 2004).3 Between 2000 and 2003, more than 400 public companies went bankrupt, including Enron, which rose to seventh on the Fortune 500 list, and Bethlehem Steel, one of the great industrial giants of the 20th century (Loomis, 2004; Serwer, 2002). Their combined liabilities reached more than $500 billion, a figure 10 times greater than the annual budget for all U.S. intelligence agencies combined (Office of the Director of National Intelligence, 2007).4 As Lewin and colleagues (2004) conclude, the empirical data clearly support the observation that “most firms are selected out” (p. 108). These findings describe organizational adaptation prospects in the best of circumstances; adaptation challenges are likely to be far greater in public-sector agencies. As Allison (1980), Moe (1989), Wilson (2000), Zegart (2007), and others have noted, private-sector firms enjoy key adaptation advantages that government agencies lack. Four are paramount. First, market competition incentivizes firms to adapt or die. Indeed, population ecology theorists argue that private-sector innovation arises between organizations, not within them: Newer, fitter firms are constantly replacing older, outdated ones through a Darwinian process of natural selection (Hannan and Freeman, 1977, 1984). But this degree of organizational churn does not exist in government. As many have observed, government agencies are notoriously hard to kill because some interest groups and 2 Note that these figures cover firm deaths each year from 1990 to 2005. Because they predate the current economic recession, they are likely to underestimate current firm death rates. 3 The three firms that remained at number one for more than a single year are General Motors, Exxon Mobil, and Walmart. Rankings are based on previous year’s revenues. 4 Intelligence budget calculations based on an Office of the Director of National Intelligence press release, which reported the first post-9/11 declassified National Intelligence Program budget: $43.7 billion for FY 2007 (Office of the Director of National Intelligence, 2007).
OCR for page 312
Intelligence Analysis: Behavioral and Social Scientific Foundations elected officials out there will always resist (Downs, 1967; Stinchecombe, 1965; Lowi, 1979; Kaufman, 1976; Lewis, 2003).5 Public-sector agencies—especially intelligence agencies—rarely fear they will go out of business.6 Instead, history has shown that policy makers usually respond to perceived government failures by creating new agencies, not eliminating existing ones. Although intelligence agencies may have other incentives to adapt, the market’s powerful imperative to change or close up shop is not one of them. These realities suggest that the benefits of competition are naturally more limited in the IC than in the private sector. On the one hand, competition can stimulate ideas, sharpen analysis, guard against groupthink and other pitfalls, and generate new ways of doing things. Yet because intelligence agencies compete without the shadow of organizational death, weak practices in one agency are likely to linger alongside better ones elsewhere. The second advantage that firms enjoy in the adaptation struggle is that their creators and employees want them to succeed (Moe, 1990; Zegart, 1999, 2007). In the business world, no one foists a new company on reluctant owners and no employee cheers silently for the day when company profits plummet. Instead, businesses are filled with organizational well-wishers who have vested interests in the organization’s continued success. Government agencies, by contrast, are created by many who want them to fail. In politics, new agencies are forced into existence by winning political coalitions who impose their will on the losers. This means that losers have a say in the new organization’s design and operation. The fragmented structure of the American political system ensures that political opponents have many opportunities to sabotage the creation of a new agency at the outset, hobbling it with all sorts of structures, rules, and requirements that hinder its performance over time (Moe, 1989; Zegart, 1999, 2007). As Terry Moe writes, “American public bureaucracy is not designed to be effective” (Moe, 1989, p. 267). Whether it’s the Environmental Protection Agency or the Office of the Director of National Intelligence (ODNI), government agencies are constrained from the start by the politics of their own creation. The third advantage businesses have when it comes to driving organizational change is managerial discretion. Subject only to minimal legal requirements, managers in private firms can determine or change their 5 David Lewis has questioned the immortality thesis, finding that 438 new agencies were created between 1946 and 1997. But to put those numbers into perspective, more businesses are born in a single day before lunch. 6 Indeed, congressional scholars have made much of Congress’s oversight powers. See McCubbins (1985); Weingast and Moran (1983); and Epstein et al. (1999). But Congress’s oversight weapons are much weaker than they appear and at times create perverse incentives, rewarding failures by granting bigger budgets, more personnel, and other corrective measures that bureaucracies value (see Moe, 1987).
OCR for page 313
Intelligence Analysis: Behavioral and Social Scientific Foundations organization’s mission; hire and fire whomever they choose; institute whatever procedures, policies, and customs they believe are necessary; and attract capital from a multitude of sources. As James Q. Wilson shows in detailed case studies that range from prisons to schools to the Central Intelligence Agency (CIA), public-sector managers are far more constrained. They can only dream of exercising this kind of discretion to shape the organization’s mission and match resources against priorities (Wilson, 2000). Fourth and finally, businesses typically have an easier time instituting major change because chief executive officers (CEOs) usually stay on the job longer than their public-sector counterparts. Although CEO tenure has declined in recent years, it still averages 7 years (Kaplan and Minton, 2006; Kelman and Myers, 2009). That’s more than twice as long as the 3.3-year median tenure of Senate-confirmed Cabinet secretaries and three times longer than the median service of deputy-secretary–level appointees in the first Bush and Clinton Administrations (Dull and Roberts, 2008). Average tenure of top intelligence officials is even shorter: Since 9/11, CIA director tenure has averaged 2 years, and directors of national intelligence (a position created in April 2005) have averaged 1.47 years. Although the Federal Bureau of Investigation (FBI) director holds a 10-year fixed term, the Bureau’s top counterterrorism position has been held by eight people since 9/11, averaging just 1 year each (Stein, 2006).7 These figures are particularly noteworthy given the fact that organization theorists consistently have found that frequent leadership turnover hurts firm performance.8 In sum, organization theory tells us that adaptation is difficult under the best of circumstances. Businesses are fortunate. They are fueled by market competition and its shadow of death, focused by a unified mission, filled with stakeholders seeking success, armed with broad managerial discretion to match resources against organizational needs, and led by senior executives who stay long enough to see major changes implemented. But even these blessings lead to failure more frequently than one might expect. 7 Since the article was printed, Arthur Cummings became executive assistant director of the National Security Branch, making him the eighth top counterterrorism official. 8 Classic early work in the 1960s and 1970s examined sports teams and found that frequent coaching turnover was correlated with poor team performance. Since the 1990s, a robust literature has found the relationship between executive tenure and firm performance to be curvilinear. Organizational performance typically rises with CEO tenure to a point, then falls as executives and organizations get stuck in outmoded thinking and practices. Importantly, Kelman and Myers (2009) note that the CEO tenure inflection point (when performance starts to diminish) is 5 years or more. This is substantially longer than the tenure of most intelligence agency heads. For turnover literature, see Kesner and Sebora (1994); Dull and Roberts (2008); Rainey and Steinbauer (1999); and Kelman and Myers (2009). For related work on institutional change and the survival of political leaders, see Bueno de Mesquita et al. (2003).
OCR for page 314
Intelligence Analysis: Behavioral and Social Scientific Foundations Insight #2: Organizational Structure Matters More Than We Think The second insight focuses on the relationship between an organization’s structure and its ability to learn. Cyert and March’s 1963 classic, A Behavioral Theory of the Firm, first introduced the idea that organizations were not fixed and rigid, but adaptive learning systems. Subsequent research was quite diffuse, but generally agreed on four important points: (1) organizational learning involves acquiring, processing, and integrating information important to the functioning of the organization; (2) organizational learning positively affects future performance (Fiol and Lyles, 1985; Levitt and March, 1988); (3) organizations learn in a host of directed and spontaneous ways; and (4) organizational structure can influence learning in profound and often hidden ways. This last point is particularly important for intelligence agencies because they are in the information learning business, confront extreme levels of uncertainty, and have faced persistent calls for structural overhaul since World War II. The list of reorganization efforts is long, including the CIA’s creation in 1947; the National Security Agency’s establishment in 1952; the consolidation of imagery into the National Imagery and Mapping Agency in 1996; the creation of the Terrorist Threat Integration Center in 2003 and its successor, the National Counterterrorism Center, in 2004; the creation of the ODNI in 2005; and repeated counterterrorism, intelligence, and national security reorganizations inside the FBI from the 1990s to the present. In each case, reformers sought to improve the IC’s performance by restructuring the organizations within it. As Hammond (2009) writes, “while many prescriptions for intelligence community ‘reform’ have proved difficult to implement, structure seems to have been subjected to reforms and reorganizations fairly often, perhaps because structural problems are seen, whether correctly or not, as more easily solved” (p. 4). Organization theorists have not settled the question of which structural arrangements are best, even in private industry. However, they have illuminated more clearly why no one best structure exists. Briefly put, organization theorists have found neutral design to be impossible; the structure of the organization itself—its hierarchy, its arrangement of subunits—affects how information is organized and what decisions result (Simon, 1976; Hammond and Thomas, 1989; Seidman, 1998). A hypothetical example illustrates the point. Imagine for a moment that you are the head of an agency, and you possess magical powers to eliminate all conceivable sources of bias so that your decisions are based solely on the information provided by your subordinates. Waving your wand, you eliminate the personal and cognitive biases of everyone in the organizational chain of command, including yourself. You neutralize the pressures of political interests and external stakeholders seeking a particular
OCR for page 315
Intelligence Analysis: Behavioral and Social Scientific Foundations outcome. You eliminate the pathologies of small-group decision making. You ensure that information does not get filtered or altered in the communications process, so each subordinate unit passes along all the information it has. You align incentives so that everyone has every reason in the world to provide “just the facts,” information that is unvarnished, untainted, and unconnected to personal or career objectives. Furthermore, let’s assume that all the information you receive is highly credible. Even in these ideal circumstances, your decision will be biased, and it may turn out to be wholly inconsistent with the data. Why? Because how you organize units in the bureaucracy determines whether the same pieces of information get concentrated as signals or dispersed as noise (Wohlstetter, 1962). Bendor and Hammond (2010) provide two simple examples that show these structural forces at work. In the first, an intelligence agency director has three bureaus monitoring terrorist groups. The director will alert the President about a possible impending attack only if at least two of the three bureau chiefs report that they are concerned about terrorist activity patterns in their domains. Bureau chiefs, in turn, operate with the same decision rule: A bureau chief will send a report expressing concern to the director only if at least two of his three subordinates raise a red flag. Reporting is determined by answering the following question: “Do you believe that the groups in your jurisdiction are intensifying their terrorist activity?” A “0” means “no,” and a “1” means “yes.” Table 13-1 shows the same data aggregated in two different structures. The first structure organizes bureaus by geography: Regions A, B, and C. Inside each regional bureau, subordinates are responsible for tracking the activities of al Qaeda-affiliated, Iran-affiliated, and unaffiliated terrorist groups. The bureau chief from region A gets signals of concern from all three subordinates (1,1,1), so he sends a report to the agency director. Region B’s bureau chief gets only one signal of concern (1,0,0), so he does not send a report to the director. Region C also has only one signal (0,1,0), so does not report a concern. In this structure, because only one of the three regional bureaus raises a red flag, the director does not alert the President. Now consider the second structure, which organizes bureaus by the TABLE 13-1 The CIA Reporting Problem Region A Region B Region C AQ-affiliated groups 1 1 0 Iran-affiliated groups 1 0 1 Unaffiliated groups 1 0 0 NOTE: AQ = al Qaeda. SOURCE: Bendor and Hammond (2010, p. 651:Table 27.2). Reprinted by permission of Oxford University Press, see http://www.oup.com.
OCR for page 316
Intelligence Analysis: Behavioral and Social Scientific Foundations type of terrorist groups they monitor. Within each bureau, subordinates track activities in different geographic regions. The bureau that monitors al Qaeda-affiliated groups receives two reports from regional subordinates (1,1,0), so it reports concern to the director. The Iran-affiliated group bureau also receives two signals from different regions (1,0,1), so it reports concern. Because two of three bureaus have reported concern, the director alerts the President. The data and decision rules are exactly the same in both structures. But because these two structures aggregate the information differently, the director warns in one case, but not the other. In the second example, Bendor and Hammond (2010) also show how hierarchies can produce counterintuitive judgments. Now an agency director wants to know whether al Qaeda-affiliated groups are more or less likely than Iran-affiliated terrorist groups to commit attacks in the near future (see Table 13-2). There are two bureaus. Bureau A’s information suggests that 20 percent of al Qaeda-affiliated groups (10 of 50) are planning terrorist attacks, while no Iran-affiliated groups are planning attacks. Bureau A therefore concludes that al Qaeda-affiliated groups are more likely to commit terrorist activities in the near future. Bureau B has different data showing that 100 percent of al Qaeda-affiliated groups (10 of 10) are planning terrorist attacks, while 80 percent (40 of 50) of Iran-affiliated groups are planning attacks. Based on these data, Bureau B also reports to the director that al Qaeda-affiliated groups are more likely to commit near-term attacks. However, when the director aggregates the data from both bureaus, she finds a very different picture: One-third of al Qaeda-affiliated groups (20 of 60) are planning near-term attacks, while two-thirds of Iran-affiliated groups (40 of 60) are planning attacks. Using the same metrics (percentage TABLE 13-2 Terrorist Activities Reports Bureau A Bureau B Terrorist Activities Planned No Terrorist Activities Planned Terrorist Activities Planned No Terrorist Activities Planned AQ-affiliated groups 10 40 AQ-affiliated groups 10 0 Iran-affiliated groups 0 10 Iran-affiliated groups 40 10 NOTE: AQ = al Qaeda. SOURCE: Bendor and Hammond (2010). Original publication included two tables: Original table: 27.3 Terrorist Activities Reports—Bureau A (p. 652) and 27.4 Terrorist Activities Reports—Bureau B (p. 652). Reprinted by permission of Oxford University Press, see http://www.oup.com.
OCR for page 317
Intelligence Analysis: Behavioral and Social Scientific Foundations TABLE 13-3 Director’s Aggregated Data from Bureaus A and B Terrorist Activities Planned No Terrorist Activities Planned AQ-affiliated groups 20 40 Iran-affiliated groups 40 20 NOTE: AQ = al Qaeda. SOURCE: Table derived from Bendor and Hammond (2010). of affiliated groups planning attacks) and the same decision rule (select the group type with the higher percentage of member organizations planning attacks), the director reaches the opposite conclusion of Bureaus A and B. She judges that Iran-affiliated groups are more likely to commit near-term attacks (see Table 13-3). As this example illustrates, data collected in subunits can lead every subunit to the same evidence-based hypothesis, even when the aggregation of data across subunits suggests the exact opposite belief. Called Simpson’s paradox, this problem is well known among statisticians and occurs when associations between variables in smaller datasets become inverted once the data are combined (Simpson, 1951). One of the more popular examples of Simpson’s paradox involves the batting averages of baseball stars Dave Justice and Derek Jeter. Although Justice had a higher batting average than Jeter in 1995 and 1996, Jeter had a higher batting average when data from both years were totaled. The reason: large differences in the number of at-bats each year (Ross, 2004). Intelligence experts, of course, have long been aware of structural dilemmas. In 1949, Sherman Kent explicitly contemplated the trade-offs between a centralized versus decentralized intelligence system as well as the relative costs and benefits of organizing units by geography or function (Kent, 1949). No arrangement, he concluded, was ideal.9 But more recent organization theory suggests that these structural problems may be even more pernicious than many realize. The Bendor and Hammond examples provide a cautionary warning: Robust analytic techniques are not enough. Organizational structures can exert enormous, unseen, and unexpected influence over how information is aggregated and what hypotheses emerge (Bendor and Hammond, 2010). Organizational structure also affects an organization’s ability to learn 9 Kent (1949) came down in favor of “the regional breakdown as far as possible,” but acknowledged that such a structure posed two problems: “how to handle matters which defy regionalization” and “how to handle those problems of a multinational nature for which the organization provides no full-time functional supervisor or coordinator” (pp. 122–123). See also Hammond (2009).
OCR for page 318
Intelligence Analysis: Behavioral and Social Scientific Foundations and improve its own performance. As Vaughan (1996) and Zegart (2007) have noted, the structure of an organization can impede its ability to adapt, even when the need to adapt is clear. The key here is specialization. In their quest for efficiency, organizations create subunits to break down large tasks into smaller ones. Each subunit becomes specialized, using particular skills, employing particular people, and developing particular knowledge so each part of the organization does what it does best. But these pockets of specialization make it difficult for one part of the organization to understand the work of another, complicate coordination by creating distance between managers and operators, and foster standardized ways of communicating and operating across organizational divisions. Although March and Simon’s (1958) classic work finds many benefits to standard operating procedures,10 more recent research finds that standard operating procedures are a double-edged sword, increasing organizational reliability but hampering innovation.11 Standard forms, automated computer systems, and reporting procedures help managers across an organization to perform the same tasks in the same ways each time. These measures, however, also weed out new ideas and stifle improvements that do not fit easily into existing forms, channels, or procedures—a phenomenon Vaughan calls “structural secrecy” (1996). Two examples show the powerful effects of structural secrecy at work. First, Vaughan’s case study of the Space Shuttle Challenger disaster finds that Morton Thiokol engineers were gravely concerned about the resilience of the shuttle’s O-ring joints in cold weather. They turned out to be right: In 1986, Challenger exploded shortly after launch because abnormally cold weather had caused the O-rings on the solid rocket boosters to fail. The night before the disaster, Thiokol’s engineers desperately tried to abort the launch. But their warnings were muted and ultimately disregarded in large part because of the National Aeronautics and Space Administration’s (NASA’s) own standard operating processes, structures, and norms. No minimum temperature launch criterion had been established, so NASA managers did not see the urgency of creating one the night before a launch. Thiokol’s crucial presentation relied on qualitative judgments from previous flights (the putty damage between the O-rings looked different in colder weather flights than others) rather than NASA’s standard “engineering-supported” technical positions that were based on quantitative analysis. Because the Shuttle program’s division of labor physically separated key 10 March and Simon argued that standard operating procedures help organizations cope with two problems: too much information and too little information. Standard procedures, they noted, simplify the task of management and provide useful feedback loops that enable managers to identify trends early enough to take corrective action before problems turn into crises. 11 For problems with standard operating procedures, see Allison (1971); Vaughan (1996); and Sagan (1993).
OCR for page 319
Intelligence Analysis: Behavioral and Social Scientific Foundations participants in different locations, the pivotal communication occurred in a three-way teleconference, with no video transmission. As Vaughan notes, “many visual cues that normally aid interpretation—such as gestures, facial expressions, body posture, activity—were unavailable.” Instead, “communication depended on individual willingness to speak to an unseen audience” (Vaughan, 1996, p. 357). Paradoxically, the very structures, rules, and technologies designed to improve organizational efficiency sabotaged NASA’s ability to learn. Zegart finds that structural secrecy also hindered the FBI’s ability to penetrate the 9/11 plot. In a 7-week period during the summer of 2001, three FBI field offices uncovered what turned out to be key clues. In Phoenix, Special Agent Kenneth Williams identified a disturbing trend, wrote a memo warning that Osama bin Laden might be sending terrorists to train in U.S. flight schools, and recommended several specific steps, including notifying other intelligence agencies. As FBI Director Robert Mueller later reflected, “You are not going to have a better intelligence product than the Phoenix memo.”12 During the same period, FBI agents in Minneapolis detained a suspicious foreign flight school student named Zacarias Moussaoui, a self-proclaimed Jihadist who wanted to fly 747s and later became the only person convicted in the United States in connection with the attacks. Third and finally, the FBI’s New York office began searching for Khalid al-Mihdhar and Nawaf al-Hazmi, two suspected al Qaeda operatives who later hijacked American Airlines Flight 77 and flew it into the Pentagon. But because the FBI was divided into 56 largely independent and autonomous field offices (one longstanding joke at the Bureau was that the FBI consisted of 56 field offices with a headquarters attached), none of the agents working these cases knew about the others. On three separate occasions in that 7-week period, the threat of a domestic terrorist attack caught the attention of someone in the FBI, but failed to trigger a broader effort to collect information, share information, or take stock of what the FBI already knew. The Bureau’s field office structure enhanced specialization—enabling individual field offices to address local law enforcement priorities—but prevented officials in one part of the organization from learning what others in the organization already knew (Zegart, 2007). In sum, organizational learning research suggests that structure matters much more than most people believe, that organizational reliability and innovation are often mutually exclusive, that managers must work outside standard operating procedures to identify obsolete practices and foster innovation, and that officials must be vigilant about monitoring how structural arrangements aggregate, or fail to aggregate, information to guard against misleading analytic judgments. 12 Personal communication, Robert Mueller, FBI, January 2007.
OCR for page 320
Intelligence Analysis: Behavioral and Social Scientific Foundations Insight #3: Internal Barriers to Organizational Change Are Powerful Social science research finds what intelligence insiders already know to be true: employees become wedded to organizational routines, thinking, norms, ideas, and identities and these attachments make change difficult (see Tinsley, this volume, Chapter 9, for discussion of these issues in greater depth). Here, a point worth underscoring is that resistance to innovation stems more from the everyday aspects of organizational life than from a few old-timers or old-thinkers. Levitt and March argue that organizational performance often falls victim to “competency traps,” which are routines that were once beneficial, but have become obsolete over time (Levitt and March, 1988; March, 1981). Avoiding competency traps requires systemic and careful work to identify and exploit “old knowledge” that still works (March, 1991; Crossan et al., 1999), “unlearn” routines that do not (Hedberg, 1981), and explore new approaches that might work better (March, 1991; Levinthal and March, 1993). For intelligence, this research suggests that improving analysis requires more than hiring talent or generating good ideas and new tools. It requires an explicit management program to identify and shed maladaptive practices, encourage the search for new and better ones, foster supportive cultures and habits, and erode counterproductive ones. Limitations The most serious limitation of organization theory is its focus on firms. As Steve Kelman (2007, p. 226) writes, “Improving government performance is a topic worthy of significant research attention, yet dramatically insufficient scholarly firepower is directed at it.” The result is that organization theory pays relatively little attention to political incentives, institutions, and power, forces that are crucial for understanding adaptation challenges in government agencies (Zegart, 2007). INSIGHTS AND LIMITATIONS OF POLITICAL SCIENCE The political science literature offers different insights and limitations for improving intelligence analysis, as described in the paragraphs below. Insight #1: Institutional Incentives Drive Behavior Although the political science literature is vast, the discipline’s dominant approach for the past 20 or 30 years has been rational choice. See Chapter 3, this volume, by Bruce Bueno de Mesquita for discussion of rational choice analysis in much greater depth and for an examination of
OCR for page 321
Intelligence Analysis: Behavioral and Social Scientific Foundations how game theoretic models offer useful analytic tools. But rational choice also illuminates the “how to make good analytic practices stick” side of the equation. Put simply, theories of rational choice focus on what makes individuals alike, not what makes them different. Rational choice theorists argue that all individuals, whatever their personalities, wants, and needs, act in predictable and systematic ways for predictable and systematic reasons: Namely, they select alternatives and conduct activities that maximize net benefits to themselves. In politics, individuals are driven by the incentives of office to maximize their political advantages. No normative judgment is implied; rational choice describes the way the political world works, not the way reformers wish it to be.13 Legislators, for example, select committee assignments that deliver benefits to folks back home because they prefer winning reelection to losing (Mayhew, 1974). Similar dynamics explain Presidential behavior. Although no two Presidents are alike, all of them wield the same powers, confront the same institutional players, seek to secure their place in history, and make decisions based on which policies produce the greatest advantages for their administration at the lowest political cost. For political scientists, outcomes stem less from the idiosyncratic personalities or beliefs of individuals, and more from the forces that transcend them (Moe, 1985, 2009). For intelligence analysis, rational choice theories remind us that leadership is not a panacea; institutional incentives frequently explain why people and organizations behave in the ways they do—for example, why constituent elements of the IC historically resisted centralization under the CIA, and why they are likely to continue resisting centralization under the new ODNI, including efforts to improve analytic practices, even now.14 At the ground level, rational choice theory suggests that bad incentives often prevent good people from improving organizational performance. A new analytic technique, for example, may produce better judgments. But getting analysts to use it requires convincing them, and their managers, that the costs of learning and using something new are worth it. Although charismatic leadership can help foster change, institutionalizing these kinds of improvements requires structuring incentives and communicating them clearly. Net career benefits—for each person involved—matter a great deal. In short, the literature suggests that making improvements stick means relying less on the force of individual personalities and more on harnessing the incentives that motivate us all. 13 Three seminal works in rational choice are Arrow (1951); Downs (1957); and Olson (1965). For an important critique, see Green and Shapiro (1996). 14 I do not mean to suggest that rational self-interest is the only reason intelligence elements might resist centralization. But it is an important and often underappreciated one.
OCR for page 322
Intelligence Analysis: Behavioral and Social Scientific Foundations Insight #2: Individual Rational Decisions, Collective Suboptimal Results Political science research also cautions that individually rational decisions can produce collectively suboptimal results. The classic example is the tragedy of the commons, where individual farmers seek to gain advantage by allowing their sheep to graze as much as possible on public lands. Yet, because every farmer has the same cost–benefit calculation, they all make the same choice. Overgrazing ensues, the fields become fallow, and everyone suffers. Current examples of tragedy of the commons problems abound. Nobody likes wasteful government spending, but every member of Congress has strong incentives to draft legislative earmarks to fund his or her district’s pet projects, leading to wasteful earmark proliferation. When the stock market starts falling dramatically, the natural reaction among nonprofessional investors (and some professional ones) is often to avoid bigger losses by selling fast. But when many respond to these incentives in the same way, the market plummets even more and losses grow. Rational behavior for one becomes detrimental for all. This same basic logic explains in part why intelligence agencies in the Pentagon and other parts of the IC historically have fought against centralized control by the Director of Central Intelligence (DCI) and its ODNI successor, even though doing so hinders the coordination and collaboration essential to intelligence success. One reason agency employees circumvent or resist central directives is that they see personal or organizational benefits to protecting their own agency’s turf and costs to ceding it. The result, however, is that the entire intelligence system suffers.15 Limitations Political science has been hampered by two key weaknesses. The first is that the field rarely treats agencies as dependent variables. Organizations are inputs to policy outcomes, not phenomena to be studied in their own right. Most political scientists are uninterested in internal organizational forces such as norms, routines, and cultures, precisely the forces that fuel bureaucratic resistance to change. Indeed, “culture” is something of a dirty word in the discipline, denoting a residual, “squishy” variable that cannot be measured clearly and that is usually employed only when all other explanations fall short. The second limitation stems from the first: Political science pays little attention to the nuts and bolts of how agencies actually work. Although public administration and political science used to be closely aligned fields, 15 Of course, there are also pathological and psychological reasons for resisting centralization, including rigid adherence to outdated agency cultures, traditions, and identities, and more general aversion to change.
OCR for page 323
Intelligence Analysis: Behavioral and Social Scientific Foundations they split decades ago. For years now, political science has considered public administration to be too practically oriented, too atheoretical, and too methodologically weak. The claims are not entirely without merit (Kelman, 2007). But the effect has been to create a yawning gap between theory and practice, and a dearth of policy-relevant political science work to inform public management. It is no coincidence that the “reinventing government” movement, which gave rise to the Clinton–Gore National Performance Review, came from practitioners instead of scholars (Kettl, 1998, 2005; Aberbach and Rockman, 2000). A WORD ABOUT THE BUSINESS MANAGEMENT LITERATURE A separate and growing body of research concerns the practical interests of managers. In the early days, the debate focused mostly on how to improve firm efficiency. Taylor’s seminal work in 1911 argued that managers’ core challenge was to institute practices that increased managerial control, reduced worker discretion, and broke down tasks into smaller and smaller pieces. Rejecting the aphorism that “Captains of industry are born, not made,” Taylor sought, as he put it, “to try to convince the reader that the remedy for … inefficiency lies in systematic management, rather than in searching for some unusual or extraordinary man” (Taylor, 1911, p. 7). Starting in the 1930s and 1940s, Harvard Business School produced an alternative “human relations” approach that found workers also needed to be motivated to be productive.16 After World War II, business programs skyrocketed, producing major changes and a growing popular orientation. In 1956, fewer than 4,000 students received a Master’s in Business Administration (MBA). By 2003, that number had topped 100,000 (U.S. Department of Education, 2005).17 In a 20-year period alone—from 1974 to 1994—the number of American universities offering MBA degrees doubled, from 389 to nearly 800 (Deutsch, 1993). Because business schools are in the business of training managers, they have an incentive to produce research that highlights the importance of leadership and the role of managers inside organizations (Kelman, 2007). Although important social science research has continued to be developed inside business schools, a cottage industry of best-selling leadership and management books has also arisen, dispensing advice to business leaders 16 In a series of famous experiments at the Hawthorne Western Electric Plant, Roethlisberger and Dickson (1949) found that worker productivity increased under any form of attention. One of the seminal theoretical works in the field is Barnard (1938). For an excellent overview of the literature, see Charles Perrow (1986). 17 That’s five times the number of students studying for Master’s Degrees in Public Policy or Public Administration (Kelman, 2007).
OCR for page 324
Intelligence Analysis: Behavioral and Social Scientific Foundations and general audiences alike.18 Despite its popularity, however, this literature has substantial limitations in improving intelligence analysis. Two reasons explain why. First, the literature assumes away nearly all of the most important constraints on government agencies. Wallace Sayre’s oft-quoted law that public and private management are fundamentally alike in all unimportant respects has fallen by the wayside (Allison, 1980). To be clear, this literature does not assert that its lessons apply well to government agencies; it neglects government agencies altogether (Kelman, 2007). General rules of thumb are drawn almost entirely from private-sector cases and are intended for private-sector audiences. Grafting these ideas from firms to intelligence agencies is difficult. For example, Jim Collins’s (2001) book, Good to Great, examines the factors that distinguish high-performing firms from average ones in the same industry. One of his key findings is personnel, or as he puts it, “getting the right people on the bus and getting the wrong people off the bus.” This advice makes good sense for companies, but overlooks important intelligence realities. In the intelligence world, anticipating who the “right people” are and how many of them you’ll need is riddled with uncertainty. The right people at one point in time (say, Warsaw Pact experts) may turn out to be the wrong people later. Conversely, some employees (e.g., Pashtu speakers) may seem relatively insignificant one day and indispensable the next. Aligning the workforce will always lag substantially behind an intelligence agency’s needs because hiring people entails undergoing a lengthy security clearance process and firing them requires dealing with onerous civil service procedures and regulations. Selecting the “right people” hinges as much on identifying intangible qualities—a willingness to embrace change and take intellectual risks, a drive to get things done, an aptitude for working well with intelligence customers and colleagues—as substantive knowledge or other measurable skills. Finally, for decades intelligence agency cultures have prized lifetime service to the mission and country, not “here today, gone tomorrow” labor markets where organizations and employees alike expect to move on as conditions warrant.19 Getting on and off the intelligence bus is not so fast or easy.20 The second limitation of this work is methodological. With some important exceptions (Collins, 2001), the popular management literature 18 Some of the best known examples are Collins (2001); Useem (1999); and Kotter (1996). 19 The demographics of today’s IC workforce raise new questions about career tenure—specifically, whether the post-9/11 generation of analysts expects more fluid career paths into and out of government and, if so, how the IC can harness top talent either through modifying retention practices or developing career paths that enable analysts to move in and out of government more easily. 20 Collins defends the applicability business practices to nonprofits and government agencies in Good to Great and the Social Sectors: A Monograph to Accompany Good to Great (2005).
OCR for page 325
Intelligence Analysis: Behavioral and Social Scientific Foundations commits many of the selection bias errors discussed by Bueno de Mesquita (this volume, Chapter 3). In general, the literature presents sweeping conclusions, nostrums, and top-10 lists based on illustrative case studies and weak causal reasoning rather than more rigorous experimental testing, surveys, or systematic research methods. Peters and Waterman’s In Search of Excellence (1981) is a classic example. The authors examine several top-performing companies, find a few things these companies have in common, and conclude that the commonalities must be the keys to success. Peters and Waterman might be right. Or they could be terribly wrong, identifying traits that are shared by most companies—successes and failures alike—and that have little or no bearing on performance. These methodological weaknesses have created a great deal of conventional management wisdom with questionable results. In 1996, John Kotter published one of the best known change-management books ever written, Leading Change. Kotter’s book contained no references, footnotes, or rigorous empirical research unless one counts occasional references to “that reminds me of a story” illustrative examples. Nevertheless, Leading Change spawned a huge change-management movement that produced thousands of articles and books. Yet in 2008, a McKinsey and Company survey of 3,199 executives around the world reported that only a third of all transformations succeeded, the same percentage that Kotter found 12 years earlier. The McKinsey study concluded, “It seems that, despite prolific output, the field of change management hasn’t led to more successful change programs” (Aiken and Keller, 2009 p. 100).21 The point here is not to criticize for the sake of criticizing. It is to shine a light on which social science research paths offer dead ends and which offer promising avenues to improve the implementation of analytic practices. In the final analysis, organization theory and political science offer some important, relevant insights. The popular management literature, however, appears far less promising for improving intelligence analysis. REFERENCES Aberbach, J. D., and B. A. Rockman. 2000. In the web of politics: Three decades of the U.S. federal executive. Washington, DC: Brookings Institution Press. Aiken, C., and S. Keller. 2009. The irrational side of change management. McKinsey Quarterly 2:100–109. Aldrich, H. 1999. Organizations evolving. Englewood Cliffs, NJ: Prentice-Hall. Allison, G. 1971. Essence of decision: Explaining the Cuban missile crisis, 1st ed. Boston, MA: Little Brown. 21 These authors go on to commit the same methodological mistakes, proffering advice about how to handle “the irrational side of change management” based largely on their own professional experiences working with companies attempting transformations.
OCR for page 326
Intelligence Analysis: Behavioral and Social Scientific Foundations Allison, G. 1980. Public and private management: Are they fundamentally alike in all unimportant respects? Proceedings of the Public Management Research Conference, November 19–20, 1979 (pp. 27–38). OPM Document 127-53-1. Washington, DC: Office of Personnel Management. Arrow, K. J. 1951. Social choice and individual values. New Haven, CT: Yale University Press. Barnard, C. 1938. The functions of the executive. Cambridge, MA: Harvard University Press. Bendor, J., and T. Hammond. (2010). Choice-theoretic approaches to bureaucratic structure. In F. R. Durant, ed., The Oxford handbook of American bureaucracy (pp. 638–665). London, UK: Oxford University Press. Bueno de Mesquita, B., A. Smith, R. Siverson, and J. Morrow. 2003. The logic of political survival. Cambridge, MA: MIT Press. Collins, J. 2001. Good to great: Why some companies make the leap … and others don’t. New York: Harper Business. Collins, J. 2005. Good to great and the social sectors: A monograph to accompany good to great. New York: HarperCollins. Crossan, M. M., H. W. Lane, and R. E. White. 1999. An organizational learning framework: From intuition to institution. Academy of Management Review 24(3):522–537. Cyert, R. M., and J. G. March. 1963. A behavioral theory of the firm. Oxford, UK: Blackwell. Deutsch, C. H. 1993. MBA programs fight for shrinking pool of students interested in business. New York Times, November 14. Downs, A. 1957. An economic theory of democracy. New York: Harper and Row. Downs, A. 1967. Inside bureaucracy. Upper Saddle River, NJ: Scott Foresman and Company. Dull, M., and P. S. Roberts. 2008. Continuity, competence, and the succession of Senate-confirmed agency appointees, 1989–2009. Presidential Studies Quarterly 39(3):432–453. Epstein, D., S. O’Halloran, R. Calvert, and T. Eggertsson. 1999. Delegating powers. New York: Cambridge University Press. Fiol, C. M., and M. A. Lyles. 1985. Organizational learning. Academy of Management Review 10(4):803–813. Green, D. P., and I. Shapiro. 1996. Pathologies of rational choice theory. New Haven, CT: Yale University Press. Hammond, T. H. 2009. Intelligence organizations and the organization of intelligence: On the problem of drawing inferences from data scattered around the bureaucracy. Unpublished manuscript. Department of Political Science, Michigan State University. Hammond, T. H., and P. Thomas. 1989. The impossibility of a neutral hierarchy. Journal of Law, Economics, and Organization 5(1):155–184. Hannan, M. T., and J. H. Freeman. 1977. Population ecology of organizations. American Journal of Sociology 82(5):929–964. Hannan, M. T., and J. H. Freeman. 1984. Structural inertia and organizational change. American Sociological Review 49(2):149–164. Hedberg, B. 1981. How organizations learn and unlearn. In P. Nystrom and W. H. Starbuck, eds., Handbook of organizational design (pp. 3–27). London, UK: Oxford University Press. Kaplan, S. N., and B. A. Minton. 2006. How has CEO turnover changed? Increasingly performance sensitive boards and increasingly uneasy CEOs. NBER Working Papers 12465. Cambridge, MA: National Bureau of Economic Research. Kaufman, H. 1976. Are government organizations immortal? Washington, DC: Brookings Institution Press. Kelman, S. 2007. Public administration and organization studies. In A. Brief and J. P. Walsh, eds., Academy of Management Annals (pp. 225–267). New York: Erlbaum.
OCR for page 327
Intelligence Analysis: Behavioral and Social Scientific Foundations Kelman, S., and J. Myers. 2009. Successfully executing ambitious strategies in government: An empirical analysis. Faculty Research Working Paper Series (#RWP09-009). Boston, MA: Harvard University, Kennedy School of Government. Kent, S. 1949. Strategic intelligence for American world policy. Princeton, NJ: Princeton University Press. Kesner, I. F., and T. C. Sabora. 1994. Executive succession: Past, present and future. Journal of Management 20(2):327–372. Kettl, D. 1998. Reinventing government: A fifth-year report card. Washington, DC: Brookings Institution Press. Kettl, D. 2005. The global public management revolution, 2nd ed. Washington, DC: Brookings Institution Press. Kotter, J. P. 1996. Leading change. Cambridge, MA: Harvard Business School Press. Levinthal, D., and J. G. March. 1993. The myopia of learning. Strategic Management Journal 14(SI):95–112. Levitt, B., and J. G. March. 1988. Organizational learning. Annual Review of Sociology 14:319–340. Lewin, A. Y., C. B. Weigelt, and J. D. Emery. 2004. Adaptation and selection in strategy and change: Perspectives on strategic change in organizations. In M. S. Poole and A. H. Van de Ven, eds., Handbook of organizational change and innovation. New York: Oxford University Press. Lewis, D. E. 2003. Presidents and the politics of agency design: Political insulation in the United States government bureaucracy, 1946–1997. Stanford, CA: Stanford University Press. Loomis, C. J. 2004. The sinking of Bethlehem Steel. Fortune 5:174. Lowi, T. J. 1979. The end of liberalism: The second republic of the United States. New York: W. W. Norton. March, J. G. 1981. Footnotes to organizational change. Administrative Science Quarterly 26 (Dec.):563–577. March, J. G. 1991. Exploration and exploitation in organizational learning. Organization Science 2(1):71–87. March, J. G., and H. A. Simon. 1958. Organizations. New York: John Wiley. Mayhew, D. 1974. Congress: The electoral connection. New Haven, CT: Yale University Press. McCubbins, M. D. 1985. The legislative design of regulatory structure. American Journal of Political Science 29(4):721–748. Moe, T. M. 1985. The politicized presidency. In J. E. Chubb and P. E. Peterson, eds., The new direction in American politics. Washington, DC: Brookings Institution Press. Moe, T. M. 1987. An assessment of the positive theory of “Congressional Dominance.” Legislative Studies Quarterly 12(4):475–520. Moe, T. M. 1989. The politics of bureaucratic structure. In J. E. Chubb and P. E. Peterson, eds., Can the government govern? (pp. 267–330). Washington, DC: Brookings Institution Press. Moe, T. M. 1990. The politics of structural choice: Toward a theory of public bureaucracy. In O. E. Williamson, ed., Organization theory, from Chester Barnard to the present and beyond. New York: Oxford University Press. Moe, T. M. 2009. The revolution in Presidential studies. Presidential Studies Quarterly 39(4):701–724. Office of the Director of National Intelligence. 2007. DNI releases budget figure for National Intelligence Program. News Release No. 22-07. Washington, DC: Office of the Director of National Intelligence. Available: http://www.dni.gov/press_releases/20071030_release.pdf [accessed June 2010]. Olson, M., Jr. 1965. The logic of collective action. Cambridge, MA: Harvard University Press.
OCR for page 328
Intelligence Analysis: Behavioral and Social Scientific Foundations Osborne, D., and T. Gaebler. 1993. Reinventing government: How the entrepreneurial spirit is transforming the public sector. New York: Penguin. Osborne, D., and P. Plastrik. 1998. Banishing bureaucracy: The five strategies for reinventing government. New York: Penguin. Perrow, C. 1986. Complex organizations: A critical reader. New York: McGraw-Hill. Peters, T. J., and R. H. Waterman. 1981. In search of excellence: Lessons from America’s best-run companies. New York: Warner. Pfeffer, J., and R. I. Sutton. 2006. Hard facts, dangerous half-truths, and total nonsense: Profiting from evidence-based management. Cambridge, MA: Harvard Business School Press. Rainey, H. G., and P. Steinbauer. 1999. Galloping elephants: Developing elements of a theory of effective government organizations. Journal of Public Administration Research and Theory 9(1):1–32. Roethlisberger, F. J., and W. Dickson. 1949. Management and the worker. Cambridge, MA: Harvard University Press. Ross, K. 2004. A mathematician at the ballpark: Odds and probabilities for baseball fans. New York: Pi Press. Sagan, S. D. 1993. The limits of safety: Organizations, accidents, and nuclear weapons. Princeton, NJ: Princeton University Press. Schlosser, J., and E. Florian. 2004. The biggest moneymakers! The best investments! The hall-of-famers and the one-hit-wonders! The triumphs, the failures, the milestones! Fifty years of … amazing facts! Fortune 5:152. Seidman, H. 1998. Politics, position, and power: The dynamics of federal organization, 5th ed. New York: Oxford University Press. Serwer, A. 2002. Breaking records—for bankruptcies: Chapter 11 is the hottest fad in business. But that’s not even the half of it. Fortune 5:22. Simon, H. A. 1976. Administration behavior: A study of decision-making processes in administrative organizations, 3rd ed. New York: Free Press. Simpson, E. H. 1951. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society (Series B), 13:238–241. Stein, J. 2006. FBI picks its seventh counterterrorism chief since September 11, 2001. Congressional Quarterly. Stinchecombe, A. L. 1965. Social structures and organizations. In J. G. March, ed., Handbook of organizations. Chicago, IL: Rand McNally. Taylor, F. 1911. Principles of scientific management. New York: Harper Bros. U.S. Census Bureau. 2009. Firm births and deaths by employment size of enterprise: 1990 to 2005. Table 739. Statistical abstract of the United States: 2009. Washington, DC: U.S. Census Bureau. Available: http://www.census.gov/compendia/statab/cats/business_enterprise/establishments_employees_payroll.html [accessed June 2010]. U.S. Department of Education. 2005. Earned degrees in business conferred by degree-granting institutions, by level of degree and sex of student: Selected years, 1955–56 to 2002–03. Table 278. Washington, DC: U.S. Department of Education. Available: http://nces.ed.gov/programs/digest/d04/tables/dt04_278.asp [accessed June 2010]. Useem, M. 1999. The leadership moment: Nine true stories of triumph and disaster and their lessons for us all. New York: Three Rivers Press. Vaughan, D. 1996. The Challenger launch decision: Risk technology, culture, and deviance at NASA. Chicago, IL: University of Chicago Press. Weingast, B., and M. Moran. 1983. Bureaucratic discretion or congressional control? Regulatory policymaking by the Federal Trade Commission. Journal of Political Economy 91(5):765–800. Wilson, J. Q. 2000. Bureaucracy: What government agencies do and why they do it. New York: Basic Books.
OCR for page 329
Intelligence Analysis: Behavioral and Social Scientific Foundations Wohlstetter, R. 1962. Pearl Harbor: Warning and decision. Stanford, CA: Stanford University Press. Zegart, A. B. 1999. Flawed by design: The evolution of the CIA, JCS, and NSC. Stanford, CA: Stanford University Press. Zegart, A. B. (2007) Spying blind: The CIA, the FBI, and the origins of 9/11. Princeton, NJ: Princeton University Press.
OCR for page 330
Intelligence Analysis: Behavioral and Social Scientific Foundations This page intentionally left blank.