Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 92
Persistent Forecasting of Disruptive Technologies 6 Evaluating Existing Persistent Forecasting Systems INTRODUCTION The committee selected three existing forecasting systems with which to compare the committee’s ideal system characteristics. Three criteria were considered for their selection. First, each of the systems attempted to forecast and track multiple technology arenas of interest to society at large (information technology, biotechnology, energy, and so on). Second, each system solicited the viewpoints of multiple experts, though the degree of openness to nonexperts varied. Third, the committee was able to access each system and meet at least once with a senior member of the staff. The Institute for the Future’s (IFTF’s) X2 system was chosen at the sponsor’s request. The remaining two systems, Delta Scan and TechCast, were chosen because they facilitated evaluation of different approaches to forecasting, presentation of the data, persistence, and openness to the public. Each of the three selected systems is described next, along with its strengths and weaknesses in comparison to an ideal system. The chapter concludes with a side-by-side analysis of all three systems according to six key characteristics. DELTA SCAN Delta Scan was developed as a tool for the Horizon Scanning Centre as part of the United Kingdom’s national persistent forecasting effort. It is the sister system to Sigma Scan, which focuses on public policy and social issues. In contrast, Delta Scan focuses on technology. Initially guided by IFTF, the system first collected science and technology perspectives from over 250 experts in government, business, academia, and communications through workshops, interviews, and wikis. Then a team from the Horizon Scanning Centre reviewed projects and programs to determine strategic priorities, with the goal of augmenting overall technological capabilities. According to Horizon Scanning Center staff member Harry Woodroof, in his presentation to the committee on October 15, 2007, the goal of the Center was to perform the systematic examination of potential threats, opportunities, and likely developments, including but not restricted to those at the margins of current thinking and planning. Horizon scanning may explore novel and unexpected issues as well as persistent problems or trends (Woodroof, 2007).
OCR for page 93
Persistent Forecasting of Disruptive Technologies The resulting technology perspectives were collected and synthesized into 100 papers by the Scanning Centre and the IFTF as they looked at potential developments in the field of science and technology over the next 50 years. The forecast comprised a short “outlook” and an Accompanying “snippet”; a set of overarching themes; and a map of the future. One example of a single-sentence outlook and its “snippet”—here, radio frequency identification—is as follows: Tracking objects is made easy by RFID Radio Frequency Identification Devices (RFID) tagging systems will probably be widely used by government, industry, retailers and consumers to identify and track physical objects by 2015. This will vastly improve corporate inventory management, increase the efficiency of logistics, reduce loss (including through theft) at all stages between manufacturer and end-user, facilitate scientific research in hospitals by making it easier to track patients and lab samples, and make mislaid objects a thing of the past as individuals will be able to “google” and hence locate objects. IFTF identified six science and technology (S&T) themes from the forecast: Small world, Intentional biology, Extended self, Mathematical world, Sensory transformation, and Lightweight infrastructure. It also identified three meta themes: democratized innovation, transdisciplinarity, and emergence. The map of the future in Figure 6-1 was created by IFTF. The results of the forecast were published both in a report and on the Horizon Scanning Centre’s Web site. They were then were used to test strategic assumptions and to identify contingency trigger points to monitor. An impact audit was performed that weighed the identified risks against existing projects and priorities. The results were used to generate new questions and create fresh strategic priorities that could be fed back into the forecasting system. See Figure 6-2 for a diagram of the Delta Scan forecasting process. Strengths and Weaknesses A strength of this forecasting platform is that its goals, process, and approach were well defined from inception. It was designed to ensure that the architecture of the underlying data store could support and produce a forecast. The forecasting process was straightforward, practical, and used established forecasting methods such as interviews with experts and professionally led workshops. The system was developed with a modest level of resources and called on professional forecasters (IFTF staff) for help. Participants were drawn from areas of expertise that corresponded to stakeholders’ priorities. The output of the forecast was clear and concise and helped to drive decision making, program direction, and resource allocation. In addition, the resulting time line and wiki are useful to future planners and forecasters. The system, though not persistent, was designed to be iterative. The system’s potential weaknesses include its lack of support for languages other than English, its emphasis on data of local rather than global origin, the exclusive use of expert views, and the single-dimensionality of the resulting forecast, which failed to offer alternatives to its vision of the future. The system was designed as an iterative platform for forecasting, but the time between forecasts is relatively long, and new signals do not immediately impact the forecast. The system requires the discrete linear processing of each step (goal setting, process design, interviews and scans, data normalization and preparation, workshops, synthesis, and audit of impact), and within each forecasting cycle all steps must be completed before a new forecast is produced. The system is therefore not designed to be persistent. While the resulting forecast was insightful, it was not particularly surprising. Topic areas
OCR for page 94
Persistent Forecasting of Disruptive Technologies FIGURE 6-1 Science and technology outlook, 2005-2055. SOURCE: Woodroof, 2007. Crown Copyright, 2005. Produced for the U.K. Foresight Horizon Scanning Centre by the Institute for the Future. Used with permission from the U.K. Foresight Horizon Scanning Centre. FIGURE 6-2 Process diagram from the Foresight Horizon Scanning Centre. SOURCE: Woodroof, 2007. Produced by the United Kingdom’s Foresight Horizon Scanning Centre and used with the center’s permission.
OCR for page 95
Persistent Forecasting of Disruptive Technologies tended to be limited to well-understood technologies, and not enough attention was paid to possible wild cards or second-order effects. The committee would like to have seen more detailed regional analysis performed in the report and a greater emphasis on identification of possible signposts and tipping points. TECHCAST TechCast is an online research system which acts as a robust tool for pooling knowledge and forecasting technology. This system is designed to be highly flexible and can be used to gather expert data from an enterprise, a specific target group, a government, or the general public. TechCast relies on the observations of volunteer experts to forecast technological advances. The experts, who are either invited or self-nominated, are regularly surveyed on topics in their area of expertise. The resulting information is then used to create reports on an ad hoc basis. TechCast was launched in 1998 by William E. Halal at George Washington University. The administrator of TechCast first scans a variety of sources to find technology that would be interesting to forecast. A wiki is produced that lists key events, data points, and trends. One hundred experts from around the world are then asked to make a prediction and comment on the predictions of other participants using a computational wiki system. Specifically, the experts are asked to predict when a new technology will be adopted, the scale of its market impact, and the expert’s confidence in his or her own forecast. It also encourages experts to explain their prediction. Figure 6-3 is a diagram of the TechCast process (Halal, 2009). This forecasting system is persistent: Each input into it automatically updates the forecast. TechCast.org has produced forecasts in 70 technology areas. Users can scan the prediction results using a wiki. Figure 6-4 summarizes the forecasts produced by the system. It describes when experts believe a new technology will reach adoption, the scale of the market, and experts’ confidence in the prediction. The computational wiki is automatically updated and its contents are added to a dashboard displaying the expert survey results, as seen in Figure 6-5. Figure 6-6 summarizes a set of TechCast.org forecasts by time, market size, and average expert confidence, while Figure 6-7 summarizes the data by sector and the expected year of mainstream adoption. Strengths and Weaknesses TechCast is a very flexible platform and can support a wide range of prediction needs. Its strength lies in its simplicity. The system is entirely wiki based and does not require physical proximity of the forecasters. The four-step process is straightforward and simple to administer and can be maintained with few resources. It is easy to use and is persistent. Forecasters get instant results, and the site is as current as the last prediction. Output is clear and easy to understand. TechCast relies on a geographically diverse set of experts for its input; approximately half the contributors are foreign. Experts are prescreened and can comment and predict across disciplines, allowing the cross-fertilization of ideas and expertise. Participation from the expert pool is good and the administrator keeps experts engaged by frequently asking for new input to new questions generated within the system. The TechCast.org Web site has some drawbacks. First of all, it is dependent on the administrators to select topics for expert commentary. The first two steps in TechCast’s process are performed by administrator, and the methodologies employed to screen and to analyze topics are not disclosed. The system’s output is also highly influenced by the composition of the expert pool, and it is unclear what the criteria are for inclusion. The TechCast.org Web site is currently English-only and does not support regional analysis. The system of communication is relatively basic, depending solely on wikis for interactions between experts. Finally, while the system analyzes background data, it does not publish the specific signals, signposts, or tipping points necessary for continued tracking of forecasts.
OCR for page 96
Persistent Forecasting of Disruptive Technologies FIGURE 6-3 TechCast online research system. SOURCE: Halal, 2009. FIGURE 6-4 Expert survey. SOURCE: Halal, 2009.
OCR for page 97
Persistent Forecasting of Disruptive Technologies FIGURE 6-5 Result of expert survey results. SOURCE: Halal, 2009. FIGURE 6-6 Longitudinal summary of forecasts. SOURCE: Halal, 2009.
OCR for page 98
Persistent Forecasting of Disruptive Technologies FIGURE 6-7 Summary of forecast results. SOURCE: Halal, 2009. X2 (SIGNTIFIC) The X2 project, under development by IFTF, is a re-envisioning of the Victorian era X-Club, designed to foster collaboration among diverse groups by a combination of social networking (including Facebook and Digg), futurism, and forecasting. It is a novel system with a greater degree of software sophistication than the other systems described. IFTF recently gave X2 a new name—Signtific—to serve as a global collaborative research platform created to identify and facilitate discussions around future disruptions, opportunities, and trends in science and technology.1 The X2 system combines workshops, an online, wiki-based platform, and ARGs to produce a forecast. It has three main activities: signal development, forecast generation, and “big story” creation. Big story creation is the X2 creators’ version of alternative future generation. Figure 6-8 shows their methodology. The first component of X2 is expert workshops. IFTF held seven workshops around the world, with each workshop having at least 15 and sometimes more than 30 expert participants. All workshops were conducted in English. Workshops were cohosted by other organizations and employed new visualization technologies such as ZuiPrezi. A typical workshop agenda included the following: headline (theme), futures of science, geographies of science, signal entry, and breakout. Figure 6-9 is the output of an X2 workshop. 1 Available at http://www.iftf.org/node/939. Last accessed May 6, 2009.
OCR for page 99
Persistent Forecasting of Disruptive Technologies FIGURE 6-8 X2 content and methodology. SOURCE: Castro and Crawford, 2008. Courtesy of the Institute for the Future. FIGURE 6-9 X2 Workshop output. SOURCE: Castro and Crawford, 2008. Courtesy of the Institute for the Future.
OCR for page 100
Persistent Forecasting of Disruptive Technologies The second component of X2 was an interactive online platform. The platform allowed experts to post signals of interest to specific subject groups. As of November 2008, 235 experts had been invited by IFTF to participate in the platform. The 28 subject groups collectively produced 697 signals, 262 forecasts, and 11 perspectives. Signals were inputted by experts (Figure 6-10), who suggested questions to be answered. Topics were tagged, and abstracts could be added by experts. Experts could comment and rate the likelihood and potential impact of any signal created. They would then use these signals to generate sample forecasts such as the following: “Growing infrastructures for ‘citizen science’ will help shape twenty-first century science.” —Hyungsub Choi, Pennsylvania “Soft technology will be a protagonist for innovations in the twenty-first century.” —Zhouying Jin, China “Bayesian networks utilized in creating more mobile social networks.” —Emma Terama, Austria From predictions captured by the X2 platform, IFTF then proceeded to generate “perspectives.” Sample perspectives included these: The Indian Ocean as a new nexus for science, Islam’s coming scientific revolution, Designing the next revolution in science, Will green chemistry be the central science of the twenty-first century? The interdisciplinary future of energy research, The transformation of science parks, and Linking innovation to manufacturing. The third component of the X2 system is the use of “thought experiments” conducted in the form of games. IFTF employed ARGs to flush out an alternative future using crowd sourcing. Players accept the premise of the alternative future and then role-play. The game is guided by “game masters” who monitor the progress of the game and generate events within the scenario. The philosophy behind the creation of the ARG is to develop engaging scenarios that reflect perspectives generated by game players and that could serve as a foundation for discussion using Web-based social media tools. IFTF felt it was equally important to develop an appropriate community architecture focused on attracting participants and game masters and to clearly communicate the promise of the game to the players. Box 6-1 shows a sample scenario, and Figure 6-11 shows a sample game screen. As mentioned previously, the games are crowd sourced. The ARG is promoted on the Internet and is open for players from around the world to play. The games are used to gain a better understanding of the impact of various perspectives and future scenarios. Strengths and Weaknesses X2 is an interesting mix of forecasting methodologies. It combines traditional Delphi approaches with innovative methods such as ARGs and expert wikis. The system is managed by experienced professional forecasters, draws on experts to flush out signals and to forecast, and then calls on the crowd (through the use of gaming) to understand impact. The workshops, which are held around the world, are attended by a diverse set of experts. The X2 platform is a persistent system that allows experts to participate at their convenience; the workshops and games are organized events. It does an excellent job of allowing experts to create categories, input signals, and discuss potentially disruptive technologies. Finally, ARG is an innovative and effective way to flush out the potential impact of an alternative future. It does a good job of attracting players from around the world to participate in an engaging role-playing exercise.
OCR for page 101
Persistent Forecasting of Disruptive Technologies FIGURE 6-10 Expert signal framework of X2. SOURCE: Castro and Crawford, 2008. Courtesy of the Institute for the Future. BOX 6-1 Sample X2 Game Scenario: Wall Street Becomes the Mechanism for Setting the Scientific Agenda and Provides the Financing It’s the year 2020, and wealthy individual entrepreneurs are at the forefront of scientific and technological innovation…. Innovation moves from institutional to individual. Who are you in 2020? How does this new model of innovation affect you? If you are a future Xoogler, what research would you fund? If you are not wealthy, does the growing influence of individuals and less government oversight concern you? What would you do? How might this change the relationship between amateur and professional scientists? SOURCE: Castro and Crawford, 2008. Courtesy of the Institute for the Future.
OCR for page 102
Persistent Forecasting of Disruptive Technologies FIGURE 6-11 Sample game screen of an ARG game used by IFTF. SOURCE: Castro and Crawford, 2008. Courtesy of the Institute for the Future. In spite of X2’s many innovations, the system had some shortcomings. Each component of the X2 process required participants to speak English, and none of the workshops were held in a language other than English. The committee was also concerned that not enough attention was paid to making sure that the participants in the workshops were diverse enough to represent the science and technology community of the region and that there was only limited participation from young scientists, researchers, technologists, and entrepreneurs. More thought could have also been given to the selection of locations for the workshops. The committee felt strongly that X2 workshops should seek out places with developing knowledge and techno clusters that had not already been surveyed to gain technology forecasts. There were also concerns that the X2 platform was not engaging enough to encourage continuous participation by the experts and that over 50 percent of the signals were generated from U.S. experts. The participation in each forecast is too sparse to allow the wiki to collect and compute statistically meaningful likelihood and impact projections. It would also be useful to specifically ask the experts to predict a realization date for each forecast as a part of gathering data for the computational wiki. EVALUATION OF FORECASTING PLATFORMS The committee’s evaluations of the three forecasting systems, summarized in Table 6-1, are seen as preliminary and subject to additional discussion in later reports. While none of the systems met all the requirements of an ideal persistent forecasting system laid out by the committee, all had elements that were valuable. The committee has already seen great potential in existing forecasting systems such as Delta Scan, TechCast, and X2 (Signtific). These projects demonstrate many of the concepts presented in this report and give the committee confidence that an open, persistent forecasting platform for disruptive technologies can be created. Any of these existing forecasting systems could serve as the foundation for a more robust system. Significant insight can be gained even with limited or partial implementation.
OCR for page 103
Persistent Forecasting of Disruptive Technologies TABLE 6-1 Initial Evaluation of Existing Forecasting Systems Tech Cast Signtific (X2) Delta Scan Data sources Partially meets. Largely from selected panel experts. Other sourcing methods and geographic sourcing not apparent. Open to invited members only. Novel data sources, collection techniques, and data audit methods not apparent. English only. Partially meets. Open system. Multiple data sources (stewards, workshops, games) and baseline data. Geographical diversity improving (workshops, Web site) but still is English only. Participant’s profiles captured. Some applications of novel data sourcing and collection techniques. Data audit methods not apparent. Partially meets. Open to access but not to contribute. Data gathered from more than 250 experts with a wide array of backgrounds through workshops and interviews. English only. Novel data sources, collection techniques, and data audit methods not apparent. Forecast methods Partially meets. Self-selected expert opinion with some predictive elements. Part qualitative and quantitative. No apparent barriers to recruiting experts. Partially meets. Forecasts and signals derived from stewards, experts, and public via workshops and games. Signals and forecasts largely qualitative. Quantitative methods not apparent. Partially meets. Principally expert opinion. Forecast supplied by the Institute for the Future. No apparent way for the public to contribute or collaborate. Forecast team Partially meets. Self-selected experts and small team. English only. No apparent barriers to recruiting experts. Methods to assess expert diversity, country or quals not published. No public participation. Partially meets. Team consists of employees, experts, and public. Public participation strong and growing in some areas—7,000 participants in the last AGR. More limited public participation in signal generation and forecasting. Evidence of third-party community development, collaboration, and initiative becoming apparent. English only. Partially meets. Forecasts supplied by the Institute for the Future. Appears to be English only. No apparent public participation. Methods to assess diversity of experts by country, culture, discipline, etc. not apparent. Data output Partially meets. Quantitative and qualitative. Each forecast quantifies estimated time of realization, confidence levels, market size, and range of dispersion. Qualitative assessments of forecast strengths and weaknesses. Visualization limited to static graphs. Unclear if data are exportable. Partially meets. Principally qualitative. Quantitative representation is limited or not apparent. Third-party access (and export capabilities) to the data on player behavior is not apparent. Largely meets. Qualitative with some quantitative. 1-5 scale assessment of impact, likelihood, and controversy. Qualitative assessment of geographical impact. Signals, enablers, inhibitors, Centers of excellence, data sources, analogies, word tags and links identified. Visualization and navigation could be strengthened. Processing tools Limited. Some enablers and inhibitors identified in forecast narrative. Processing done by the expert community. Diversity of experts unclear. Other processing tools (dashboards, data visualization, signal and link processing) are not apparent. Limited. Some enablers and inhibitors identified by the community. Diversity of the community processing the data appears to be improving. Other processing tools (dashboards, data visualization, signal and link processing) are not apparent. Limited. Signals, enablers, inhibitors identified but no apparent way to automatically measure progress toward thresholds. Diversity of community processing data unclear. Other processing tools (dashboards, data visualization, signal and link processing) are not apparent.
OCR for page 104
Persistent Forecasting of Disruptive Technologies Tech Cast Signtific (X2) Delta Scan System attributes Partially meets. System is persistent but not open. Bias mitigation processes not apparent. Degree of scalability unclear. English only. Intuitive system with limited communication tools. Partially meets. System is persistent and open. Bias mitigation processes not apparent. Degree of scalability unclear. English only. Additional communication tools could improve usability. Limited. Open to access but not to participate. System not persistent (last updated in 2006). Bias mitigation processes not apparent. Degree of scalability unclear. English only. Additional communication tools could improve usability. Summary Partially meets. TechCast partially meets the attributes of the persistent forecasting system with strengths in ease of use and in quantifying the probability of occurrence, impact, and forecast timing. Could improve by broadening methods of data sourcing, utilizing more forecasting techniques, incorporating multiple languages, diversifying the forecast team (including the public), and strengthening data output and processing tools. The forecast produced is a single view of the future synthesized by the system operator. Partially meets. X2 partially meets the attributes of the persistent forecasting system, with strength in openness, qualitative data capture, multiple data sources, and multiple forecasting methods. Could improve by adding multiple language support, strengthening quantitative forecasting methods, processing tools, and visualization techniques. The system lacks consistent participation from users and forecasters. Forecast suffers from inadequate preparation of the initial problem set and definition of the scope of the forecasting mission. Partially meets. Delta Scan partially meets the attributes of the ideal forecasting system with particular strength in the robustness of the data forecast output. The system could be strengthened by being persistent and including additional data sources, forecast methods, public participation, native language support, and better processing tools. REFERENCES Castro, Cesar, and Mathias Crawford. 2008. X2: Threats, opportunities, and advances in science & technology, Presentation to the committee on November 6 by the Research Director for the Institute for the Future. Halal, William E. 2009. Forecasting the technology revolution, Presentation to the committee on August 3 by the President of TechCast, LLC. Woodroof, Harry. 2007. Delta Scan, Uses in U.K. government, Presentation to the committee on October 15 by the Leader, Delta (S&T) Scan, Horizon Scanning Centre, Government Office for Science, United Kingdom.