Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 57
Persistent Forecasting of Disruptive Technologies 5 Ideal Attributes of a Disruptive Technology Forecasting System One key objective of this study is to develop a forecasting approach that helps identify potential disruptions caused by technology that would have massive impact. Current methodologies for technology forecasting, such as those described in Chapter 2, are generally incapable of predicting extreme scenarios, especially those in which the most beneficial or catastrophic events occur. The committee believes that forecasters must look to the tail of the distribution curve of a technology’s probability of emergence (at the time of prediction), since it is the locus of technological developments that are usually ignored and therefore hold the potential for surprise, as can be seen in Figure 5-1. Many disruptions emerge when seemingly unrelated resources, people, events, and technologies converge. The ubiquity of the Internet, improvements in cost-efficiency of data storage, increasing processing power, and the globalization of trade and knowledge have converged to provide the opportunity for new tools, forums, and methods that will help identify emerging disruptions. These tools may allow scenarios to be tested in new ways. This chapter aims to give a comprehensive description of the high-level goals and system characteristics for an ideal persistent forecasting system (the process flow is illustrated in Figure 5-2). An overview of the key tenets of a forecasting system is followed by a discussion of the relevant characteristics and examples of information sources feeding the forecasting system. Given the increasing size, scale, diversity, and complexity of data sets, information processing is of paramount importance. Data must be structured in such a way that automated systems can aid human analysts in recognizing new correlations and relationships. Given multidimensional forecasting output, selecting appropriate visualization schemes can also help human analysts to process complex output more quickly and creatively. Finally, as for any complex system, using and maintaining a persistent forecasting system yields a number of postprocessing and system management considerations, described in the final section of the chapter. TENETS OF AN IDEAL PERSISTENT FORECASTING SYSTEM Given the breadth, complexity, and dynamism of this project, it is useful to establish desired attributes to guide the development of a persistent forecasting system. Specifically, the system should have the following attributes: Persistence, Openness and breadth, Proactive and ongoing bias mitigation,
OCR for page 58
Persistent Forecasting of Disruptive Technologies FIGURE 5-1 Probability of technology emergence. Robust and dynamic structure, Frames of reference for historical comparisons, and Ease of use. Persistence Persistence is one of the most important criteria to consider when designing a system for forecasting disruptive technology. Because most existing forecasts are developed over a short and finite time, they fail to incorporate signals that emerge after their creation and are therefore usable for only a short time. A key goal of a persistent system is to continuously improve the forecast based on new data, signals, and participant input. A persistent forecast can be built to serve many different customers, providing a continuously active and up-to-date forecast. Openness and Breadth No single group has the human, capital, or intellectual resources to imagine every possible disruptive scenario, capture every signal, or have access to all critical data. The persistent forecasting system must therefore be open to the widest possible participation. The more broad-based the system, the more likely it will be to generate many alternative futures and more extreme versions of the future, which often predict the most disruptive outcomes. The committee believes that the entities running a persistent forecasting system need to be transparent and partner-friendly. This will build trust and yield compelling content and incentives that encourage broad and ongoing participation from a diverse (on every vector) group of participants. The information derived from an open, persistent, crowd-sourced forecasting system can serve as a useful starting point for other classical approaches of forecasting, which in turn produce data and information to be fed back into the open system. Some forecasting methodologies can be employed to further investigate specific areas of interest and provide a deeper understanding of scenarios used to engage a targeted group for expert opinion. This approach uses both an iterative process, in which new ideas and forecasts are generated by crowds, and concept refinement, performed by experts. This feedback approach also exploits the strengths of other methodologies.
OCR for page 59
Persistent Forecasting of Disruptive Technologies FIGURE 5-2 Conceptual process flow for the persistent forecasting system.
OCR for page 60
Persistent Forecasting of Disruptive Technologies Engage Both Crowds and Experts Experts are typically better than novices at judging the importance of new signals in an existing forecasting system (Enis, 1995). With the currently available platforms (X2, Techcast, and Deltascan), experts generally provide high-signal and low-noise forecasts. However, academic research (Önkal et al., 2003) suggests that experts are not necessarily better at making forecasts than a crowd. Experts may not catch the full range of alternative solutions from adjacent fields outside their areas of expertise or from the reapplication of technologies developed to solve a different problem. Paradoxically, the narrowness of the knowledge specificity required to achieve expert status can invalidate forecasts generated by experts alone (Johnston, 2003). Thus, it is the committee’s belief that blending input from experts and crowds will lead to better forecasts of disruptive technologies. The goal of public participation, or crowd sourcing, in a forecasting system is to cast a wide net that gathers a multitude of forecasts, signals, and opinions. This is especially important as technology innovation becomes more diverse and geographically diffuse in its approaches and as regional variations of technology applications flourish. Collaboration technologies, especially those that leverage the power of the Internet, can be used to discover expertise in unexpected places.1 Prediction markets, alternate reality games (ARGs), and relevant online communities are disseminating crowd-sourced methods. Managing Noise in an Open System Increasing the diversity of participants will increase the richness of a forecast. Nevertheless, open and public forecasting systems also present challenges to their operators. One of the challenges is the noise and distractions generated by such systems. There are many different strategies for reducing the noise in crowd-based systems. Some systems limit participation to prescreened invitees. Screening is especially useful if a forecast seeks the opinions of a specific audience based on topic, region, or demographics (i.e., young European postdoctoral fellows studying quantum computing). Another approach is a completely open and public site, with fillers to select those with appropriate reputation, expertise, and credentials. Crowd-sourcing sites can use moderators who are themselves experts to monitor, moderate, and augment the forecasts and discussion. These moderators can be either internal staff members or volunteers from the community of users discovered through the Web. Incentives for Contribution It is important that a persistent forecasting system be not only open but also effective and relevant. For a system to generate enough signals and forecasts to be of value and to have adequate global representation, the operators must have a large number of diverse and active participants to cover the range of topics. This suggests that it should have adequate incentives (both financial and nonfinancial) to secure the ongoing participation of a diverse user base, to access technologically and socioeconomically impoverished contributors, and to persuade owners of proprietary data to donate (or sell) their data. Spann and Skierra, as well as Servan-Schrieber and colleagues, suggested that the most effective incentives might be monetary or nonmonetary, depending on circumstances (Spann and Skiera, 2003; Servan-Schreiber et al., 2004). Incentives could utilize elements of gaming (competition), reputation, and financial rewards. Attention must be paid to the cultural appropriateness of the incentives used to secure reliable and valid data. Much of the world’s population resides in collectivistic and hierarchical societies, where information is much more likely to be shared with one’s own group2 than with strangers (Triandis, 1995). An in-group is made up of people sharing similar interests and attitudes, producing feelings of solidarity, community, and exclusivity.3 An out-group is made 1 Available at http://itstrategyblog.com/whats-better-crowd-sourcing-or-expert-sourcing/. Last accessed May 6, 2009. 2 In sociology, an “in-group” is a social group for which an individual feels loyalty and respect, usually because he or she is a member , while an “out-group” is defined as a social group for which an individual feels contempt, opposition, or competition. Available at http://en.wikipedia.org/wiki/Ingroup and http://en.wikipedia.org/wiki/Outgroup_(sociology). Accessed on April 14, 2009. 3 Available at http://dictionary.reference.com/browse/ingroup. Last accessed May 6, 2009.
OCR for page 61
Persistent Forecasting of Disruptive Technologies up of people outside one’s own group, who are considered to be alien.4 In collectivistic cultures, deception has been found to be more common among out-group members than in Western cultures (Triandis et al., 2001). In such cultures, financial incentives to share information might be less useful than symbolic incentives developed through contacts, with mutual commitment and obligations assured through connections within local networks. Proactive and Ongoing Bias Mitigation Bias in a forecast can create blind spots resulting in surprise. One way to reduce it is to continually ensure that ample data from a balanced and wide range of sources are used to create the forecast. Types of bias and techniques for bias mitigation are discussed in detail in Chapter 4. Robust and Dynamic Structure The world is a rapidly evolving, highly complex place with many interdependencies. This makes it difficult for decision makers to clearly define the parameters of forecasting disruption. Determining the cause and effect, as well as the precise timing, of interactions between technology and society is fraught with uncertainty as these interactions are nearly impossible to define in advance. Further, simplistic correlations are typically not useful, and the course of human events is frequently altered by unforeseen, random, and high-impact events. Therefore, the persistent forecasting system must be dynamic, flexible, and robust enough to embrace and incorporate great uncertainty, complexity, multiple perspectives, and sometimes unclear strategic imperatives. Provisions for Historical Comparisons Disruptions occur when trends are interrupted or when historical correlations or linkages among assets, people, or topics diverge. Looking for these disruptions requires that a system have adequate data to track historical and existing trends, as well as linkages to help users spot disruptions and discontinuities. One useful frame of reference is to consider how disruptive technologies have developed and emerged historically. It would be useful to review the life cycle of a specific disruptive technology from concept, development, introduction, and adoption through maturity and obsolescence. Identifying key signposts, measurements of interest, and tipping points in the past would help us to recognize evolutionary patterns of development and some of the key inhibitors or enablers of disruptive technologies. Another useful frame of reference would be to look back at the grand challenges in history and analyze how innovation and technologies were applied to overcome them. Understanding how initiatives and technologies failed to produce the kinds of disruptions originally hoped for by their creators can be just as important as understanding those that succeeded. The system should include sufficient historical data to help researchers and forecasters to identify indicators of a disruption. Comparing such indicators against a baseline measurement of environmental and economic conditions may increase efforts to find and develop a new disruptive replacement for existing technologies. For example, understanding the threshold cost of gasoline and the necessary price performance ratio of alternative fuels and batteries would be useful for tracking a disruptive shift in propulsion systems for automobiles. Ease of Use To attract and maintain the broad participation of third parties, the system must have a robust set of processing and communication tools that are easy to use and accessible for partners, analysts, and participants alike. Function and ease of use for as diverse a population as possible must be designed into the system from the beginning. This should include a means for attracting users in languages other than English and a user-friendly graphical user interface. 4 Available at http://dictionary.reference.com/browse/outgroup. Last accessed May 6, 2009.
OCR for page 62
Persistent Forecasting of Disruptive Technologies INFORMATION COLLECTION Understanding which data to collect or gain access to is an important step in building and maintaining a useful system. Gathering data for its own sake is neither useful nor productive and can result in information overload. Considerations for Data Collection Traditionally, there was a desire to collect all the necessary data over a fixed period and then to create a forecast once those data have been captured and ingested. However, this method can create epistemological bias (Faber et al., 1992) because it assumes ex ante that only a limited set of data connections and conclusions is possible. As for complex adaptive systems (CASs), it is not possible at any single point in time to identify all linkages and causalities or to model the consistent rational behavior of all the players and forces. Therefore, it is important to design a data repository that can be initialized with the relevant historical and current data sets and then populated with ongoing, real-time data collections (Jonas, 2006). This repository is used to create ongoing and on-demand forecasts. The new forecasting system requires decision makers to have a different mental model of such a technology forecast. Instead of using a technology forecast to predict a single most likely scenario, the decision maker uses the system to understand the signals, signposts, and emerging picture of multiple alternative futures over time. Table 5-1 illustrates how the new system would compare to traditional forecasting models using a puzzle analogy. TABLE 5-1 Puzzle Analogy Type Traditional Forecasting New Forecasting Visual Analogy A single puzzle, pieces are known Multiple puzzles, pieces distributed at random, media is inconsistent, tools required are unknown, assembler is blindfolded Metaphor One-time disruptive forecasting Persistent disruptive forecasting Context A single best guess of a most likely future Multiple alternative futures Collection It is possible to gather all the pieces We don’t need all the pieces—just enough to see emerging patterns or pictures Time There will be time to run a dedicated processing step Processing must be ongoing, as we cannot anticipate disruption
OCR for page 63
Persistent Forecasting of Disruptive Technologies TABLE 5-2 Availability of Key Data for the Persistent Forecasting System Readily Available Partially Available Not Available Digital Ingest Negotiate, collaborate, or create proxies Create proxies Nondigital Negotiate digital rights (when necessary); digitize or manually input into machine-readable form Negotiate, collaborate, or create proxies Create proxies Handling Data Overload The cost of information collection, storage, processing, and dissemination has dropped consistently over the history of mankind. From stone engraving to cuneiform, scribes, the printing press, and now to the Internet, large data warehouses, and server farms, a society’s ability to store, process, and disseminate information has increased dramatically (Blainey, 2002). Amazon’s Elastic Compute Cloud (EC2) and Simple Storage Service (S3) and Google’s App Engine exemplify the lowered cost of information storage and processing power. According to the 2003 Berkeley SIMS study How Much Information?, approximately 5 exabytes (5 million terabytes) of new information was created in 2002.5 This is equivalent to approximately 800 megabytes per person, or 500,000 libraries the size of the U.S. Library of Congress print collection. One article estimated that Google stored just 0.02 percent of the world’s information in 2006.6 Despite continued dramatic improvements in computational power, storage, and network connection speeds, there is still too much data stored in too many locations and too many data formats with various levels of accessibility (privacy, cost, language) and fidelity to allow simply blindly loading data into a persistent forecasting system. The more complex the data correlation, the more time and computational power needed to identify relationships. A data set of size n looking for two-factor correlations could theoretically take approximately n2 computations, while a three-factor correlation would take approximately n3 computations. For example, a dataset with 1 million elements could theoretically take 1 trillion computations to analyze all of the two-factor correlations, while a process of trying to find three-factor correlations could take up to one quintillion (1018) calculations. Therefore, it is important that the data prioritization and structuring processes are done before the data are gathered. Persistent and systematic data gathering is an essential step in assembling a persistent disruptive technology forecasting system. The good news is that the information available on the Internet grows with every passing moment. However, this suggests that the forecasting system will also need to consider data source prioritization and filters to eliminate duplicative or less relevant sources. Further, it is important to complement online data extraction with classical methods of information collection to supplement and validate online data in an efficient and judicious manner. Collection Limitations Some pieces of information critical to the persistent forecasting system may be in a format not easily extractable or translatable, may be proprietary, or may not be available electronically or at all, as shown in Table 5-2. The operators of the persistent forecasting system should, accordingly, be creative in improving the accessibility of viscous data or in creating proxies (substituted information sources) where data are not otherwise available. Depending exclusively on digital data collections is not sufficient. Indeed, a significant portion of the world’s population remains offline, for reasons including poverty, lack of electronic accessibility, legacy systems, or a desire to remain anonymous. As a result, classic data-gathering techniques such as workshops, article reviews, document collection, and first-hand interviews or reporting are still critical. These efforts should be pursued on a cost-effective basis to serve areas where there is a known lack of information, a need for bias mitigation, or when scope broadening is required. 5 A 3-year study started in 2008 to determine how much information there is in the world is being undertaken at the Global Information Industry Center at the University of California, San Diego. Available at http://hmi.ucsd.edu/howmuchinfo.php. Last accessed May 6, 2009. 6 Available at http://www.knowledgebid.com/media/blog/google-225b-and-02-percent/. Last accessed October 23, 2008.
OCR for page 64
Persistent Forecasting of Disruptive Technologies There are any number of reasons why data might be only partially available, including rights to use, fees, language, format, digital or analog, and matters of privacy and ownership. The operators of a persistent forecasting system should consider a number of ways to access partially available data. When relevant data are in a foreign language, operators might consider employing context and translation tools or use crowd-sourcing techniques to translate. Translation tools, while not perfect, continue to improve, making data in foreign languages more available. The quality of existing commercial and open source extract, transform, and load (ETL) tools continues to improve, making it increasingly possible to extract information from different (particularly unstructured data) formats. In other cases, it might be possible to negotiate better terms for commercial databases or to create or join open collaboration communities. Finally, it may be necessary to digitize and manually enter data if it is not already available in electronic form. It is anticipated that there will be a number of subject areas where information is not readily accessible, probably owing to gaps in coverage. Statistical analysis might show which certain regions of the globe systematically report less information than expected. It is anticipated that there will be similar underreporting for technologies that show promise but for which there remains insufficient publicly available information. In other cases, it may be impossible to find or access the data for reasons of confidentiality or other factors. In these situations, the persistent forecasting system operators should consider utilizing proxies. For instance, the pricing of certain key raw materials may not be publicly available. In this situation, it may be necessary to negotiate with the supplier(s) of that raw material for access to information. Failing that, operators of the persistent forecasting system might consider creating a proxy consisting of key production inputs of the raw material, collaboration communities, or predictive networks. Key Characteristics of Information Sources In selecting information sources to use in a persistent forecasting system, the operator must first ask whether the data have the following characteristics (sources that score above average in several characteristics may be more useful than those that score very high in only one or two characteristics): Can be evaluated anywhere, Updated regularly, Relevant, Quantifiable, Sources are accurate and attributed, and Source has a good reputation. Jonas points out that data warehouse operators must keep track of data source characteristics, but should be careful to not exclude data because of poor quality characteristics.7 Changes in the quality of information and the emergence of new data sources may in and of themselves be a sign of pending disruption. Globally Evaluable An inherent requirement for the persistent disruptive forecasting system contemplated by this report is the ability to evaluate information on a global basis, making it important that the information gathered be globally sourced. A quantitative measurement of this characteristic would be a chi-squared distribution ranking the amount of information created by a country in comparison to a number of factors including but not limited to that country’s overall population, its university population, gross domestic product (GDP), number of patent applications, published scientific papers, and R&D investment. 7 Personal communication with Jeff Jonas, Chief Scientist of the IBM Entity Analytics group and an IBM Distinguished Engineer. See information available at http://www.research.ibm.com/theworldin2050/bios-Jonas.shtml. Last accessed May 6, 2009.
OCR for page 65
Persistent Forecasting of Disruptive Technologies Regularly Updated To forecast persistently, continuous sources of new information are necessary. In general, data may be persistently or periodically available or available once only. A data set collected once is less useful than a data set that will continue to be generated over time. Such persistence will allow for the data to be tracked, evaluated, and trended over time. The persistence of an information source would be measured as the standard deviation of quantity or the value of data sets created year by year. Comparative Relevance Relevance is difficult to determine, particularly when attempting to forecast future events that are disruptive in nature. It must be thought of as a relative measure for allocating scarce analytical capabilities. It is a challenging issue because some of the best signals are those not likely to be viewed as acceptable or mainstream. Nevertheless, one potential measure of relevance is how it affects a potentially disruptive technology, directly or indirectly. Comparative Quantifiablilty Given the massive amount of data that exists and the ever-growing number of tools and methods for analyzing the world around us, there are few realms that are not quantifiable. This characteristic, similar to relevance, must be used relatively. For example, use of the words “semipermeable membranes are important” by a single writer is not quantifiable. However, an increase in the number of writers using those words, from 1,750 in 2009 to 2,000 writers in 2010 in a community of 20,000 experts, could be a very useful quantitative measure. Accurate and Attributed Sources When gauging the accuracy of a source, it is important to ascertain where the data in the source came from. Anonymous information may be correct, useful, and valid, but care must be taken to ensure that it is not deliberately misleading. There are also cases where a large number of sources agree but are incorrect (e.g., observations of the Loch Ness monster). The accuracy of the data from a source can be measured by understanding the basic attributes of the source and tracking its output over time to determine if the earliest data were correct. Source Reputation One measure of a source’s trustworthiness is its reputation. Reputation can be assessed by examining the number of times information from a source is cited in other credible works. Google’s PageRank is based on the principle of citation analysis,8 which is not, however, perfect. If it is not applied appropriately, citation analysis can be biased, perpetuating current thinking and conventional wisdom. Openness, peer review, and certification by a trusted third party are also important to minimize bias and engender trust in a forecasting system. An information source should be considered more trustworthy if it allows outsiders to challenge and review its methodology for collection, data hygiene, and management. Potential Sources of Information There are many sources of potentially useful data available to the public. They include the Beige Book,9 government reports from around the world, and other reports issued by nongovernmental organizations (NGOs), 8 More information available at http://en.wikipedia.org/wiki/Bibliometrics. Last accessed May 6, 2009. 9 The Summary of Commentary on Current Economic Conditions, commonly known as the Beige Book, is published eight times per year. Each of the 12 Federal Reserve Banks gathers anecdotal information on current economic conditions in its district from the reports of Bank and Branch directors and interviews with key business contacts, economists, market experts, and other sources. The Beige Book summarizes this information by district and sector. Available at http://www.federalreserve.gov/fomc/beigebook/2009/. Last accessed August 10, 2009.
OCR for page 66
Persistent Forecasting of Disruptive Technologies TABLE 5-3 Questions Posed to Data Sources at Different Phases of Inspection Phase 1 Questions Phase 2 Questions Is this field currently experiencing disruption? Who stands to gain and who is the most at risk should a disruption occur? Who is gaining in the field and who is being displaced? Could technology play a role in disrupting this field? Which factors are driving this disruption? Are technology investments being made that target this field? If so, Which disruptive technology (if any) is enabling this disruption? Which can be used to mitigate or amplify the disruption? How much is being invested? How many research personnel are pursuing the technology of interest? How many man-hours are expended in this area each week? Each month? Each year? Which nations are pursuing these potentially disruptive technologies most aggressively? Which natural resources are enablers of these disruptive technologies? Which phrases, terms, or other indicators are used to describe a potential disruption of this field? Which new phrases, terms, or other indicators are associated with this disruptive technology and disruptive field? research organizations, corporations, trade organizations, universities, open source consortiums, online S&T repositories, and S&T Web sites. Given the broad range of technologies that are the focus of this report and the need to anticipate disruptive technologies and events, including those that might be considered to have a low probability of occurrence but high potential impact, it is essential that collectors maximize their collection resources effectively, utilizing traditional as well as more novel sources of data. Furthermore, information must be gathered with an eye to answering two similar yet distinct questions: (1) How do you identify fields that are being disrupted as well as fields that could be prone to disruption (Phase 1 question)? (2) How do you monitor how far those fields have progressed toward disruption (Phase 2 question)? Some of the questions that might be posed to these data sources are listed in Table 5-3. The information sources described in the remainder of this section may be of interest and could be useful to a disruptive technology forecasting system. Trade Associations and Magazines According to Wikipedia, there are over 7,600 trade associations in the United States.10 Las Vegas, Nevada, is estimated to have hosted at least 624 trade shows in 2008, with over 2,000,000 total attendees.11 If each trade association generated just one piece of searchable literature outlining market and technological trends, together they would constitute a robust source of information. Further, as trade associations are an outcome of broad, industry-wide initiatives, they tend to present information in a manner that is particularly well suited to the goals of a persistent forecast. Universities and Cooperative Research Centers The 1,150 accredited colleges and universities in the United States collectively educate almost 5,000,000 students per year.12 The National Science Foundation has sponsored 71 industry and university Cooperative 10 Available at http://en.wikipedia.org/wiki/Industry_trade_group#cite_note-0. Last accessed October 23, 2008. 11 Count of calendar events as of August 1, 2008, from the Las Vegas Tourism Bureau Web site. Available at http://www.lasvegastourism.com/index.html. Last accessed October 23, 2008. 12 Available at http://www.aacu.org/about/index.cfm. Last accessed October 24, 2008.
OCR for page 67
Persistent Forecasting of Disruptive Technologies Research Centers, 47 of which remain active.13 These centers foster collaboration between universities and industry. To monitor and understand the inner workings of these research centers would require a significant, but not extraordinary, amount of work and configuration during the initial construction stages of a persistent forecasting system. It is imperative to gather information about universities and centers of excellence because they are hubs for many global technology clusters. A simple understanding of college and university graduation rates would provide insight into areas of growing importance. Another approach is to track the number of times a program or facility is referred to on the Internet.14 Most universities and colleges throughout the world post information about their institutions on the World Wide Web. Braintrack.com, an online directory of the world’s universities and colleges, lists and provides links to more than 10,000 universities in 194 countries. Similar support should be made available for institutions carrying out research in disruptive technologies. Academic Papers Nearly 3 million scientific papers were published in the United States across all fields between 1996 and 2006.15 Worldwide, this number was nearly 8 million over the same time period, resulting in 800,000 papers per year that could be analyzed and incorporated in a persistent system. Using Zipf’s law or other power-law distributions of paper citations, it should be possible to observe how the importance of a paper changes as references to it increase over time (Gupta et al., 2005). Financial Data There is no shortage of financial information. Financial data are so pervasive that many novel methods of finance are back-tested and applied to historical financial information, such as cotton prices in the 1800s, rather than tested using current information (Mandelbrot, 1963). The total value of all globally traded public securities is estimated to be $51 trillion.16 The United States alone has over 17,000 public companies.17 Each of these public companies produces detailed quarterly financial reports and public disclosure statements, as well as significant amounts of data on pricing and volume of trades. Public equities are not the exclusive source of potentially valuable information; unlisted securities may hold more information if it is available. The National Venture Capital Association says that $7.4 billion was invested in early stage companies through 977 transactions during the second quarter of 2008 alone.18 Similar information is available on European and Australasian venture capital from the European Private Equity and Private Venture Capital Association and the Australian Private Equity and Venture Capital Association. However, as is the case with public securities, financial information on private/venture capital is more easily gathered on domestic than on overseas markets.19 Commercial Databases There are many commercially available databases. One example is ProQuest, which has archived and made searchable over 125 billion digital pages of information and works with over 700 universities to distribute over 60,000 dissertations and theses a year on their online database. Questia allows access to over 2 million books, articles, and journals, while competitor HighBeam Research has over 60 million peer-reviewed papers represent- 13 Available at http://www.nsf.gov/eng/iip/iucrc/directory/index.jsp. Last accessed October 24, 2008. 14 International Colleges and Universities is a Web site that lists 8,750 accredited colleges and universities around the world and ranks the top 200 based on Internet references. Available at http://www.4icu.org/. Last accessed May 6, 2009. 15 Available at www.thomson.com. Last accessed November 11, 2008. 16 As of August 18, 2008. Available at http://en.wikipedia.org/wiki/Stock_market. Last accessed October 24, 2008. 17 Available at http://answers.google.com/answers/threadview?id=543247. Last accessed October 24, 2008. 18 Available at www.nvca.org. Last accessed October 23, 2008. 19 Available at http://www.evca.eu/ and http://www.avcal.com.au/. Last accessed October 24, 2008.
OCR for page 81
Persistent Forecasting of Disruptive Technologies high-impact events into various types of CAS models (such as in virtual worlds, ARGs, MMORPGs, online social networks, or approaches using techniques to generate mass scenarios). For instance, it might be useful to create existing models of the physical world (or use existing ones) and systematically test various futures, injecting random high-impact events such as economic calamity, sudden resource scarcity, natural disasters, war and conflict, or a scientific breakthrough. From this exercise, the stakeholders should be able to identify key tipping points, system fragilities, unrecognized linkages, critical interdependencies, or resource scarcities that may signal areas of technological obsolescence or increased pressure for technological innovation. It is imperative that stakeholders analyze the potential scope of disruptions and put in place a portfolio of investments in advance of a potential event so that if the event does occur, stakeholders are able to increase the number of alternative reactions and outcomes. Using this approach, decision makers will likely be better prepared to mitigate the effects of surprise with proper planning and resource allocation. Text Mining With almost 80 percent of global information stored in unstructured text (see the discussion on processing unstructured data), mastery of text mining will be a critical enabler for a persistent forecasting method that reduces the probability of unlikely events.47 Without text mining, it would be extremely difficult and expensive to organize, structure, and load RDBs and RDFs. Text mining opens up the potential for harvesting information from the Web on a massive scale and structuring the data for analysis. The disciplined, continuous scanning required to enable a persistent forecasting effort is made feasible by text mining systems. Fortunately, such efforts are not novel, with the United Kingdom having gone so far as to establish the National Center for Text Mining (NaCTeM).48 Data Relevancy Testing One method of preventing information overload is the application of relevancy tests. Presenting only data that could lead to impactful disruptive events would be an obvious way of managing information load. This suggests that when a potential disruptive future event is identified, system operators will need to utilize backcasting techniques to identify enablers, inhibitors, and force drivers. Once identified, data feeds could be monitored for each of these factors, with information presented to the user only when certain threshold levels are hit. Storing Queries Pioneered by Jeff Jonas (2006) as part of work first done to help understand cases of fraud in Las Vegas casinos, Non-Obvious Relationship Awareness (NORA) uses protocols similar to RDF analysis; however, it is dependent on a well-characterized initial question set, which is then used to count events or activities and then determine the likelihood that they are related. Specifically, this system allows queries to be stored as part of the data set and remain active for the life of the database. This creates an opportunity for the data and the query to coincide and signal the event to an end user. In effect, the query is acting like an agent receiving confirming or disconfirming feedback. Pattern Recognition Tools Pattern recognition tools are also critical to finding early weak signals in complex data sets. With appropriate visualization tools, human analysts are capable of picking up some patterns in large, complex sets of data. Augmenting human analysis with computer pattern recognition can greatly enhance pattern recognition performance, especially in large dynamic data sets. The field of pattern recognition is well established. Specialized pattern recognition tools and techniques will need to be developed over time, particularly as new disruptions occur. As part of the review and assessment phase, the persistent forecasters will need to develop pattern learning sets in their 47 Available at http://en.wikipedia.org/wiki/Intelligent_text_analysis. Last accessed October 23, 2008. 48 Available at http://www.ariadne.ac.uk/issue42/ananiadou/. Last accessed October 24, 2008.
OCR for page 82
Persistent Forecasting of Disruptive Technologies priority areas of focus. The International Association for Pattern Recognition would be a good place to start for identifying cutting-edge and relevant techniques that might be applicable for disruptive technologies.49 Link Analysis Tools Link analysis and monitoring are powerful tools for identifying signals of a potential pending disruption. Rapid variations in the rate of change of the strength, frequency, or number of linkages relative to the historical baseline or the status quo can serve as a tangible warning that the likelihood of a disruption is increasing. This requires that the persistent forecasting system establish baseline data for key areas of focus and the driving forces described earlier. Unforeseen or underappreciated (from a risk standpoint) linkages, interdependencies, and self-organizing factors manifest themselves when the dynamics of a key driver changes. This is the hallmark of a complex adaptive system (CAS). The persistent forecasting system operators should consider including a baseline of historical reference data in priority areas and robust link analysis tools to identify changes in linkages, including the establishment of new links, the breakage of established links, and changes in the strength or number of links. Using link and visualization tools like those used by the Visual Thesaurus, one can imagine the relationship between words and terms50 and other entities. These tools are also used to visualize social and information networks. Many link analysis tools are well established and available commercially. OUTPUTS AND ANALYSIS Identifying the appropriate tools and creating the right processes and engines by which information is analyzed are only the first steps to ensuring the success of a persistent forecasting system. The information must also be presented in a clear and consistent manner to facilitate analysis. The analysis is typically facilitated via a range of visualization tools that provide alternative dynamic representations of the processed data. Visualization tools help humans to be more efficient and effective in recognizing patterns in massive data sets. The operators of the persistent forecasting system should incorporate a robust set of visualization tools. Signal Evaluation and Escalation In a signal escalation process, the forecasting system uses methods that allow signals to be analyzed individually by both experts and the crowd. An expert looks at a signal and concludes the analysis by recommending that the signal be either ignored or escalated. Individual expert analyses and recommendations should then be collected and posted in public for all to see. Creating an ongoing conversation among experts and a crowd with multiple viewpoints should be one of the goals of a persistent forecasting system, with the aim of stimulating still further feedback and analysis. A signal should be escalated even if only a single expert recommends it—that is, escalation should not depend on consensus. To deter consensus interpretation of signals, experts from different disciplines should be encouraged to review signals separately. The expert(s) recommending escalated analysis should also suggest the appropriate escalation type. For example, should another expert with expertise in another discipline review the data? Should the signal be placed into a virtual world, a role-playing game, or some other method of testing and learning? Does the signal spur more questions? In this last case, such questions should be input into the system as a persistent query. Visualization Forecasts provide important data on potential futures to decision makers and stakeholders but must be represented in a clear and informative way in order to be useful. Computers organize and represent data in a vastly different way than human beings: Representations suitable for a computer algorithm are often not suitable for 49 More information available at http://www.iapr.org/. Last accessed August 24, 2009. 50 Available at www.visualthesaurus.com. Last accessed October 24, 2008.
OCR for page 83
Persistent Forecasting of Disruptive Technologies FIGURE 5-6 Example of a heat map. SOURCE: Google Analytics. humans and vice versa. If a decision maker cannot understand a model or the solution produced by an algorithm, then the algorithm, model, and solutions are effectively useless.51 The human mind can quickly process complex information received visually. Computer-generated visual representations and interaction technologies provide the tools for users to see and interact with large volumes of information at once. Visual analytics build on this ability to facilitate the analytical reasoning process (Thomas and Cook, 2005). With forecasting methodologies such as trend analysis, TRIZ, influence diagrams, and prediction markets, visualization tools can produce a graphical model of the progression (both past and potential future) of technology and the factors that might have a direct effect on the development, adoption, and applications of technology. Existing Visualization Tools There are well-known methods of charting, graphing, and presenting information that have been successful for decades. Pie charts, bar graphs, and scatter plots are the standard output of Microsoft Excel, Google spreadsheets, statistical analysis software (SAS), and Mathematica. However, with the dramatic increases in computing power over the past 50 years, there are many new ways to visualize information. Maps, long one of the richest methods for visualizing data, may now be combined with data sets in a number of novel and visually empowering manners. Heat maps, like those used by Google Analytics and SmartMoney, are popular examples of visualizing clusters of related data (see Figure 5-6). Other graphics tools such as geo mashups, link diagrams, transaction maps, and 51 Christopher Jones, Visualization and Optimization, Bellingham, Wash.: Kluwer Academic Publishers, 1996, p. 3.
OCR for page 84
Persistent Forecasting of Disruptive Technologies time-line plots are widely used by analysts for investigatory analytics. Graphs of network nodes, subways, gene paths, page views, and other data-intense subject matter have pushed the boundaries of visualization and shown great potential for expansion into new areas.52 Ideal Visualization Dashboard Dashboards are executive information systems that present data in summary formats. They leverage visual analytics and intuitive user interface designs to produce interactive and easy-to-read overviews of complex data holdings. It is believed that the ideal dashboard will have several key visualization and analytical attributes. Handle large amounts of data. Visualization tools must allow the users to evaluate mass quantities of data or granular data quickly and with equal ease. For example, it might be useful to have a heat map that allows the user to look at the entire universe of data at a glance to identify outliers or concentrations of activity. This capability should be combined with interactive tools that allow the user to further investigate specific data points of interest. Some examples of such capabilities are HealthMap,53 which monitors disease alerts on a global map, and FINVIZ,54 a financial visualization site designed to support stock traders that monitors stock market activity overlaid on a heat map. Vary timescales. The dashboard should have capabilities that allow users to manipulate or change views on a variety of different timescales. Changes that might be missed in the shortterm might become more obvious on a longer timescale. Use global representations. The dashboard should visually represent to the users potential disruptive activities occurring throughout the world. Data sets and real-time data feeds can be combined55 with maps to provide geocontext for the data. This is especially important given the relationship of disruptive technologies to techno clusters.56 The dashboard should use geospatial visualizations like those developed by Google Earth and the Environment Remote Sensing Institute (ERSI). Macro- and user-defined alerts. The dashboard should also have alerts that are both macro- and user-defined. For example, FINVIZ generates tables that show the performance of macromarket indicators as well as stocks that are performing at the extremes or that have hit certain threshold signals (see Figure 5-7). Macromarket indicators include total advancing/declining stocks, total new highs/new lows, and total stocks above/below the moving average. FINVIZ also has tables that show stocks that have tripped widely monitored threshold triggers such as unusual volume activity, most volatile, overbought, oversold, top gainers, top losers, stocks hitting new highs or new lows, stocks with large numbers of insider transactions, and stocks with upcoming earnings announcements. The FINVIZ site also includes several technical default thresholds such as stocks hitting trendline support or trendline resistance and stocks that are in an up channel or a down channel. These are all examples of indicators that are of interest to particular types of financial advisors. A similar system could be built for forecasters to track signals and signposts of potentially disruptive technologies. Allow for search and real-time filtering. The dashboard should include tools for search, filtering, alerting, and analytics. Some examples might include (1) keyword search, (2) filtering based on user-defined variables, and (3) user-defined analytics to recognize changes in relationships between variables. 52 An excellent discussion of data visualization can be found at the Many Eyes site, which is part of IBM’s Collaborative User Experience Group. Available at http://services.alphaworks.ibm.com/manyeyes/home. Last accessed May 6, 2009. 53 Available at http://healthmap.org/en. Accessed on August 7, 2009. 54 Available at http://finviz.com/. Accessed on August 7, 2009. 55 A mashup is a Web application that combines data or functionality from one or more sources into a single integrated application. More information available at http://en.wikipedia.org/wiki/Mashup_(web_application_hybrid). Last accessed May 6, 2009. 56 A techno cluster is a high-technology-oriented geographic concentration of interconnected technology businesses, suppliers, universities, and associated institutions. More information available at http://en.wikipedia.org/wiki/Business_cluster. Last accessed May 6, 2009.
OCR for page 85
Persistent Forecasting of Disruptive Technologies FIGURE 5-7 Finviz.com dashboard. SOURCE: Courtesy of FINVIZ.com, Financial Visualizations. Support user communities. The dashboard should have tools that allow for the formation of user communities and provide tools to capture and monitor signals of interest from these communities. An ideal persistent disruptive technology forecasting system might include global maps showing patent filings, venture capital investments, natural disasters, raw materials supply chains, and political stability. These indicators might be color-coded to show improving/deteriorating conditions in each of the major indicators. Indicator-specific dials would allow the user to drill down into subjects of interest, especially outliers. For instance, if an earthquake occurs near the mine in Chile that supplies a significant portion of the globe’s lithium needs, natural disaster and key raw materials triggers might be jointly tripped that would flash as a hot spot on a global map (see Figure 5-8). The end user would then drill down at the hot spot, where the earthquake and raw material data would be in a state of deterioration. Drilling down another level would reveal companies, technologies, and products that could be adversely impacted by the event and also reveal alternative sources of supply. Chronological dials could be moved backward and forward in time to see how much lithium material is available in the supply chain and how it has changed over time in order to assess how long it will take for the shortage of lithium to impact end users. The dashboard should also include tables showing thresholds that have been recently triggered. An example might include using a tag cloud (see Figure 5-9) or a link diagram to track the appearance of a new word from an established research domain in a scientific journal or paper, or a list of new links in an online social network, suggesting that relevant experts/analysts evaluate the meaning of the trigger.
OCR for page 86
Persistent Forecasting of Disruptive Technologies FIGURE 5-8 Screenshot of Havaria Information Services AlertMap. SOURCE: Courtesy of RSOE, http://visz.rsoe.hu/alertmap/index2.php. FIGURE 5-9 Tag cloud generated for the scientific revolution using the Many Eyes Beta Site for Shared Visualization and Discovery by IBM. SOURCE: Courtesy of Many Eyes. More information available at http://manyeyes.alphaworks.ibm.com/manyeyes/page/Tag_Cloud.html.
OCR for page 87
Persistent Forecasting of Disruptive Technologies POSTPROCESSING AND SYSTEM MANAGEMENT CONSIDERATIONS Review and Reassess Building the ideal persistent forecasting system will be an iterative process that may take years to mature. The committee believes the best way to improve the system over time is to systematically conduct internal and external reviews of processes, tools, and personnel. A persistent forecasting system comprises a combination of technology, processes, and people. While this report focuses on the technology, it is just as important to consider the other two components when creating a persistent forecast. These reviews should be conducted on a regular basis, at least annually. Internal and external reviews should be done separately and by rotating personnel. The forecasting system should also have a robust set of tools for receiving feedback from the general public and from internal staff and experts. When a disruption does occur, whether predicted or not, the operators of the persistent forecasting system should consider backtesting to see if it could have been detected earlier with different techniques or personnel. Expert Interpretation of Automatic Output It is important to note that while computer modeling, systems, and heuristics have grown increasingly sophisticated they still lack the ability to assert insights. Computers are extremely efficient and effective at performing calculations, analyzing, and finding basic patterns. Search engines and databases are capable of analyzing hundreds of millions of pages of information per second. They are perfect for performing extremely high-speed repetitive operations, but for all of their speed they are still not able to mimic the performance of the human mind for understanding complex associations and patterns. The human mind is remarkably skilled at finding solutions through intuition and sophisticated questioning. This ability of humans to understand patterns and problems without previous experience or empirical knowledge separates humans from computers. Computer analytical and alerting systems are powerful tools that can be programmed using human rules to analyze large data sets. They can search the Internet for specific patterns, generating dashboards of information for human analysis. That information can be harnessed by the user through the Web to extract the wisdom of crowds (Surowiecki, 2004) through crowd sourcing and expert-sourcing. The idea behind creating an open, persistent forecasting system is to bring together the power of computer networks and the wisdom of human networks. Backtesting Looking back from 2009, it is easy to see what some of the most important technological themes over the past 30 years have been: ubiquitous communications (Internet, cell phones, wireless devices) and pervasive computing (PCs, cloud computing, server farms), digitization (iPods, DVDs, digital books), globalization of the supply chain (information technology, overnight transport), stealth (military applications), energy efficiency (automobiles, low-energy appliances, computing), clean energy (wind and solar) , bioengineering (genetic research, pharmaceuticals, regenerative medicine), and micro- and nanoscale manipulation (MEMs, nanomaterials). In 1979, would those same themes have been so easily anticipated? How would previous forecasts have identified these themes and their disruptive impact? Venturing further back in history to 1905, one could witness the battle between two rival technologies to power the world’s automobiles. In 1906, a steam-powered vehicle set the land speed record, reaching 127 miles per hour (see Figure 5-10). Steam-driven vehicles were quieter, and their underlying technology was more widely known, allowing for easier service (Pool, 1997). However, steam was not to win. At that time, the gasoline engine was just an emerging technology, but the increasing facility in drilling for petrochemicals and the advent of mass production techniques would create one of the most disruptive technologies of the twentieth century. Why did forecasters at that time not see how important this battle was and the long-term technology impact of the gasoline engine? Backtesting the forecast methods employed by the system over time is important, and comparing past forecasts to their actual outcomes will be a critical aspect of learning from these efforts and refining the system. Backtest-
OCR for page 88
Persistent Forecasting of Disruptive Technologies FIGURE 5-10 Stanley Steamer Rocket setting the land speed record in 1906 of 127.659 mph. SOURCE: Available at http://patentpending.blogs.com/patent_pending_blog/images/stanley_steamer.jpg. ing is a good way of evaluating the effectiveness and efficiency of forecasting tools and the quality of collection sources. System Management The persistent forecasting system operators and their infrastructure must also meet a minimum set of characteristics, detailed below. Financial Support Strategies The fact that the system is persistent suggests that long-term financial support and sustainability are critical requirement and that the system cannot be cost-prohibitive to either create or sustain. In general, the allowable cost should be proportional to the perceived economic benefit of a forecasting system that would lead to improved decision making. A number of strategies can be used to finance persistent systems. One strategy is to have one or more sponsors commit to an ongoing sponsorship of the system. Typically, there are anchor sponsors that make a long-term financial commitment. These sponsors can be government agencies from one or more governments, corporations, foundations, academic institutions, or nonprofit organizations. Another approach is to incorporate the activity into an internal organization that provides funding. A third approach is to build an organization (either for profit or not for profit) for the exclusive purpose of operating the system as a stand-alone enterprise. The organization could be funded by revenues from selling forecasting services and publishing forecasts, direct donations, or by foundational or governmental support. Secure System Infrastructure The persistent forecasting system requires an infrastructure robust enough to protect data against outages, malicious attack, or intentional manipulation. This is especially true for a persistent forecasting system that has been designed for global collaboration. The data system should support the ability to be parsed and analyzed across multiple quantitative and qualitative vectors simultaneously, without compromising the underlying raw data. The technology infrastructure needs to be highly scalable to support a large user base with an adequate amount of bandwidth and processing power so as not to discourage active and ongoing participation. Additionally, a robust back-up and recovery architecture must be in place.
OCR for page 89
Persistent Forecasting of Disruptive Technologies Because persistent forecasting systems will likely contain many complex data sets, the data structure must be dynamic and scalable. The underlying infrastructure (storage, processing, networking, and power) must be designed for speed, capacity, and redundancy. Commercial examples of such infrastructures include Google’s BigTable and Amazon’s SimpleDB.57 Information Integrity The persistent forecasting system must be free from manipulation. Maintaining robust auditing and monitoring tools is an important feature of a trusted forecasting system. Furthermore, it should gather, process, and disseminate information in as close to real time as possible. Certain simulations and other procedures may require advanced data management tools such as snapshotting58 and cache coherency. Data authenticity is itself an important topic. Data should not be discarded simply because it cannot be authenticated as genuine beyond repudiation. In fact, early signals of a potential pending disruption are often incongruous with the known order and therefore difficult to authenticate or corroborate. Therefore, the committee believes that although the ideal system should consider data authenticity, it should not require that data be authenticated. Information integrity59 and credibility are among the most important attributes of any forecasting system. Together they improve the legitimacy of the forecast and increase the confidence of decision makers in allocating resources. Information integrity can be threatened by bias. Sourcing and assessing data from multiple perspectives (countries, languages, cultures, and disciplines) can significantly reduce the adverse impact of bias, as discussed in Chapter 4. Concealed manipulation of data can undermine information integrity, so that an ideal system should consider using a variety of techniques and algorithms to identify when, how, and why data have been changed to reduce undetected alteration. An immutable audit log should be built into the forecasting system to track the source and flow of all of the data and the forecast. It would be a tamper-resistant record of how a system has been used and would invite everything from when data arrives, changes, and departs, to how users interact with the system. Each event is recorded indelibly, and no one, including the database administrator with the highest level of system privileges, could alter the record. Immutable audit logs enable users to track information flows and analyze the way each prediction was made.60 Because each prediction can be traced to its source, the creditability of a forecast can be established based the source of the data and how it was processed and interpreted. To allow for an understanding of the timeliness, relevance, and legitimacy of data streams, information integrity processes should, at a minimum, do the following: Keep track of data characteristics, including time, source, format, geography of origin, and language; Measure accuracy of the data source; Determine source reliability (has the data contributed by the source been accurate in the past?) or measure confidence in the data (in addition to asking participants to make a prediction, ask them how confident they are in the prediction); Ascertain data availability (persistent, periodic, or one-time availability); Perform regular audits of the system; Determine if the data were altered; and Provide strong controls for privacy and rights management. By tracking the global information characteristics of the data surrounding a potentially disruptive technology, forecasters can assess the maturity of the technology space (how frequently is the technology of interest discussed?), 57 Available at http://labs.google.com/papers/bigtable.html and http://aws.amazon.com/simpledb/. Last accessed May 6, 2009. 58 A snapshot is a copy of a set of files and directories as they were at a particular point in the past. Available at http://en.wikipedia.org/wiki/Snapshot_(computer_storage). Last accessed May 6, 2009. 59 “Information integrity” is the trustworthiness and dependability of information. It is the accuracy, consistency, and reliability of the information content, processes, and systems. Available at http://www.infogix.com/information_integrity_defined. Last accessed July 17, 2009. 60 Available at http://jeffjonas.typepad.com/jeff_jonas/2007/11/found-an-immuta.html. Last accessed, July 16, 2009.
OCR for page 90
Persistent Forecasting of Disruptive Technologies the rate of diffusion (spread of “new mentions”), and the potential impact (breadth and depth of mentions—how many cultures, disciplines, and languages?). Resource Allocation and Reporting Once a potential disruptive technology has been identified, key decision makers will want an assessment of the likelihood of the disruption, its probable impact, factors that could accelerate the disruption, and factors that could inhibit it. Further, they will also want to know key interdependencies or fragilities in the set of events that could lead to a disruption. This information should provide the decision makers with the information that they require to allocate resources that will shape the impact of the disruption when it does occur. Further, decision makers will need to be periodically updated on the status of potential disruptions. Therefore, operators of the persistent forecasting system should create a disruptive technology assessment report for each new potential disruption identified, in addition to regular updates. REFERENCES Published Anders, George. 2008. Predictions of the past. The Wall Street Journal. January 28. Available at http://online.wsj.com/article/SB120119993114813999.html. Accessed April 8, 2009. Blainey, Geoffrey. 2002. A Short History of the World. Chicago, Ill.: Ivan R. Dee, Publisher. Enis, C.R. 2005. Expert-novice judgments and new cue sets: Process versus outcome. Journal of Economic Psychology 16(4): 641-662. Faber, Malte, Reiner Manstetten, and John Proops. 1992. Toward an open future: Ignorance, novelty, and evolution. Pp. 72-96 in Ecosystem Health: New Goals for Environmental Management, Costanza, Robert, Bryan G. Norton, and Benjamin D. Haskell, eds. Washington, D.C.: Island Press. Greenfield, P.M. 1997. Culture as process: Empirical methodology for cultural psychology. In Handbook of Cross-Cultural Psychology, J.W. Berry, Y.H. Poortinga, and J. Pandey, eds., Boston, Mass.: Allyn and Bacon. Günther, Isolda de Araujo. 1998. Contacting subjects: The untold story. Culture and Psychology 4(1): 65-74. Gupta, Hari M., Jose R. Campanha, and Rosana A.G. Pesce. 2005. Power-law distributions for the citation index of scientific publications and scientists. Brazilian Journal of Physics 35(4A): 981-986. Johnston, Rob. 2003. Reducing analytic error: Integrating methodologists into teams of substantive experts. Studies in Intelligence 47(1): 57-65. Available at https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol47no1/article06.html. Last accessed May 8, 2009. Jonas, Jeff. 2006. Enterprise intelligence: Conference proceedings from TTI/Vanguard. Available at http://jeffjonas.typepad. com/jeff_jonas/2007/03/enterprise_inte.html. Last accessed October 23, 2008. Kennedy, John F. 1961. Special message to the Congress on urgent national needs. John F. Kennedy Presidential Library and Museum. Available at http://www.jfklibrary.org/Historical+Resources/Archives/Reference+Desk/Speeches/JFK/003POF03NationalNeeds05251961.htm. Last accessed on August 10, 2009. Kuechler, M. 1998. The survey method: An indispensable tool for social science research everywhere? American Behavioral Scientist 42(2): 178-200. Mandelbrot, B. 1963. The variation of certain speculative prices. Journal of Business 36: 394-419. Önkal, D., J.F. Yates, C. Simga-Mugan, and S. Öztin. 2003. Professional vs. amateur judgment accuracy: The case of foreign exchange rates. Organizational Behavior and Human Decision Processes 91: 169-185. Pareek, U., and T.V. Rao. 1980. Cross-cultural surveys and interviewing. In Handbook of Cross-Cultural Psychology, vol 2, H.C. Triandis and J.W. Berry, eds. Boston, Mass.: Allyn & Bacon. Pool, Robert. 1997. Beyond Engineering: How Society Shapes Technology. New York: Oxford University Press. Schwartz, Peter. 1991. The Art of the Long View. New York: Currency Doubleday. Servan-Schreiber, Emile, Justin Wolfers, David M. Pennock, and Brian Galebach. 2004. Prediction markets: Does money matter? Electronic Markets 14(3): 243-251. Shirkyon, Clay. 2008. Gin, television, and social surplus. Available at http://www.herecomeseverybody.org/2008/04/looking-for-the-mouse.html. Last accessed October 24, 2008.
OCR for page 91
Persistent Forecasting of Disruptive Technologies Spann, Martin, and Bernd Skiera. 2003. Internet-based virtual stock markets for business forecasting. Management Science 49(10): 1310-1326. Surowiecki, J. 2004. The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies, and Nations. New York: Random House, Inc. Tauberer, Joshua. 2008. What is RDF and what is it good for? Available at http://www.rdfabout.com/intro/?section=contents. Last accessed October 23, 2008. Thomas, J.J., and K.A. Cook. 2005. Illuminating the Path: The Research and Development Agenda for Visual Analytics. Richland, Wash.: National Visualization and Analytics Center. Available at http://nvac.pnl.gov/agenda.stm. Last accessed May 8, 2009. Triandis, Harry C. 1995. Individualism and Collectivism. Boulder, Colo.: Westview Press. Triandis, Harry C., Peter Carnevale, Michele Gelfand, Christopher Robert, S. Arzu Wasti, Tahira Probst, Emiko S. Kashima, Thalia Dragonas, Darius Chan, Xiao Ping Chen, Uichol Kim, Carsten de Dreu, Evert Van de Vliert, Sumiko Iwao, Ken-Ichi Ohbuchi, and Paul Schmitz. 2001. Culture and deception in business negotiations: A multi-level analysis. International Journal of Cross Cultural Management 1(1): 73-90. Vanston, John H. 2003. Better forecasts, better plans, better results. Research Technology Management 46(1): 47-58. Weber, E., C. Hsee, and J. Sokolowska. 1998. What folklore tells us about risk and risk taking: Cross-cultural comparisons of American, German, and Chinese proverbs. Organizational Behavior and Human Decision Processes 75(2): 170-186. Unpublished V.S. Subrahmanian, 2008. “Scalable text mining.” Presentation to the committee on May 29, 2008.