The Federal Aviation Administration (FAA) has supported the development of a quantitative model that estimates the task load on controllers created by air traffic activity in each of the more than 750 sectors of the nation’s en route airspace. The model uses traffic operations and flight-planning data to simulate the traffic activity in each sector. It then associates with this traffic the specific controller tasks that must be performed, computes and assigns a time to perform each task, and calculates the total time spent by controllers on the modeled tasks. FAA has been using the model’s task load output to estimate the number of controllers required to work the traffic in each sector, an estimate known as “positions to traffic” (PTT).
FAA asked the National Academies to convene an expert committee to examine and offer advice where appropriate for improving (a) the overall technical approach of task-based modeling, (b) input data and processes used for modeling traffic activity, (c) tasks and methods used to assign task times, and (d) means for validating model assumptions, parameters, and output. In addressing this charge, the committee was asked to be cognizant of the “overall tradeoffs made due to data availability” and to consider the “adaptability of the approach to reflect changes in the tasks of controllers as their roles evolve over time.”
Findings from the assessments in the previous chapters are provided next, including those relevant to the use of task load output for estimating PTT. These findings are followed by recommendations for model improvements.
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 58
5 Findings and Recommendations The Federal Aviation Administration (FAA) has supported the develop- ment of a quantitative model that estimates the task load on controllers created by air trafﬁc activity in each of the more than 750 sectors of the nation’s en route airspace. The model uses trafﬁc operations and ﬂight- planning data to simulate the trafﬁc activity in each sector. It then associ- ates with this trafﬁc the speciﬁc controller tasks that must be performed, computes and assigns a time to perform each task, and calculates the total time spent by controllers on the modeled tasks. FAA has been using the model’s task load output to estimate the number of controllers required to work the trafﬁc in each sector, an estimate known as “positions to traf- ﬁc” (PTT). FAA asked the National Academies to convene an expert committee to examine and offer advice where appropriate for improving (a) the overall technical approach of task-based modeling, (b) input data and processes used for modeling trafﬁc activity, (c) tasks and methods used to assign task times, and (d) means for validating model assumptions, parameters, and output. In addressing this charge, the committee was asked to be cognizant of the “overall tradeoffs made due to data avail- ability” and to consider the “adaptability of the approach to reﬂect changes in the tasks of controllers as their roles evolve over time.” Findings from the assessments in the previous chapters are provided next, including those relevant to the use of task load output for estimat- ing PTT. These ﬁndings are followed by recommendations for model improvements. 58
OCR for page 58
Findings and Recommendations 59 FINDINGS Task-Based Approach The results of task-based modeling can be a valuable source of objective information for workforce planning, and FAA’s current model is a marked improvement over previous models that did not account for trafﬁc com- plexity. The basic structure of the CAASD model, in which trafﬁc activity is simulated and controller tasks and task times are associated with trafﬁc, represents a logical approach to estimating task load. Trafﬁc Modeling Compared with simple trafﬁc counts, the simulations of trafﬁc in the CAASD model provide a more complete picture of both the volume and nature of trafﬁc activity in the en route domain. The trafﬁc activity is modeled in sufﬁcient depth and resolution to enable reasonable approx- imations of trafﬁc complexity and associated controller tasks. CAASD can check the model results against records of actual traffic activity to improve the trafﬁc modeling capabilities. Task Coverage The nine tasks in the model appear to be representative of the main R-side services that must be performed to work trafﬁc—although whether the speciﬁc claim that 90 percent of R-side tasks are covered has not been well established. Compared with the other eight modeled tasks, the monitor- ing task is treated in the most confusing manner and is difﬁcult to connect with trafﬁc activity. A simpler and more transparent means of estimating monitoring time deserves consideration. While the model’s coverage of R-side tasks may be sufﬁcient for trafﬁc capacity analysis, the omission of all tasks performed by the associate controller makes its task load out- put inadequate for estimating PTT. Task Time Derivation Many of the task times in the model are derived from a separate model- ing process known as Goals, Operators, Methods, and Selection Rules (GOMS). For seven of the nine modeled tasks, GOMS is used to derive
OCR for page 58
60 Air Trafﬁc Controller Stafﬁng in the En Route Domain task times. The other task times are developed through consultations with subject matter experts. None of the task times is derived from the observation and analysis of controllers performing tasks in the field or in experiments. The GOMS-derived times are based largely on expert judgment and are only loosely validated against a limited set of task per- formance data obtained from human-in-the-loop (HITL) experiments conducted for other purposes. The use of GOMS to derive many task times, coupled with reliance on expert judgment for validating these modeled times and for estimating many others, raises serious questions about the accuracy of the model’s task load output. Computation of Task Load Summing all of the time spent on tasks may be the most practical approach for computing total task load. However, adding one task time to another does not account for the possibility—and real-world probability—that some tasks are performed concurrently. The additive approach also does not account for the possibility that the time it takes to perform a speciﬁc task may vary depending on the level of trafﬁc activity and the number of controllers working the sector. Taking these possibilities into account may not have meaningful effects on the modeled task load. Examining their potential effects, however, is important for making this case. Conversion of Task Load to PTT FAA and model developers have sought to compensate for the absence of D-side task load by employing various processes that infer total task load to facilitate conversion to PTT. All of the PTT conversion methods used, including the current method of fuzzy logic modeling, exhibit the same fundamental ﬂaw—they imply an understanding of total task load without ever identifying the unmodeled tasks, much less measuring the time it takes to perform them. The conversions rely on experts to deter- mine thresholds and to assign complexity weightings to the unidentiﬁed and unmodeled tasks. The D-side task loads implied by these thresholds and weightings are not validated, nor can they be in the absence of any empirical data on task performance. On the whole, the use of the fuzzy logic modeling to infer task load adds little more than spurious precision
OCR for page 58
Findings and Recommendations 61 to the PTT estimates while complicating and reducing the transparency of the modeling process. Validation Modeled trafﬁc activity can be checked for accuracy through comparisons with records of actual trafﬁc. In contrast, validating PTT estimates is more challenging since there is no external measure of stafﬁng requirements against which the accuracy of the estimates can be judged. Analyzing stafﬁng records is of limited value since the main purpose of PTT model- ing is to ﬁnd out when stafﬁng levels can be better aligned with trafﬁc demand. The main method by which model developers have sought to assess PTT estimates is by presenting them to groups of experts, often consisting of individuals who manage and staff the en route centers. Yet such checks can suffer from the same shortcoming that limits the value of comparisons with stafﬁng records—the potential for bias toward existing stafﬁng practice. Because PTT estimates cannot be assessed through direct observation, all of the model’s key assumptions, processes, and parameters must be well justiﬁed and validated. A lack of data on task performance precludes vali- dation of the task times constructed from GOMS and the task complexity weightings used in the fuzzy logic conversion method. The deﬁciencies of these two modeling processes go well beyond parameter validation, as explained earlier. Yet the lack of empirical data on task performance has hindered validation throughout the modeling process, from assessing key assumptions about tasks being performed sequentially and at a ﬁxed pace to characterizing the tasks handled by the associate controller. Data Availability and Model Adaptability In the study charge, FAA asked the committee to be cognizant of trade- offs that must be made because of limited data availability, which presum- ably refers to the cost and complications of obtaining task performance data. FAA also asked for advice on the model’s adaptability to reﬂect changes in controller roles and tasks over time. Many of the ﬁndings cited earlier point to a need for a ﬁrmer empir- ical basis both for evaluating the structure of the model and for estimating
OCR for page 58
62 Air Trafﬁc Controller Stafﬁng in the En Route Domain the values of the parameters used in it. By and large, the model was devel- oped and has been evaluated with heavy reliance on the insights and opinions obtained from subject matter experts and facility personnel. More objective and quantitative task performance data are clearly needed, not only for developing the model parameters and evaluating the task load output but also for including more controller tasks in the modeling of PTT. The committee recognizes that gathering such data from operational and experimental settings will require more resources and access to controllers, which may present budget and labor relations issues. Although such cost implications were not examined in this study, it must be pointed out that there is a cost in model credibility from not obtaining such data. This cost is manifested in many ways throughout the current model, from the added opaqueness caused by fuzzy logic modeling to the excessive reliance on expert opinion and judgment for model development and validation. Whether FAA is committed to taking this data-gathering step will pre- sumably depend on its assessment of the cost trade-offs and its plans for using the model for a long time and for other possible purposes. Not knowing these plans, the committee nevertheless believes FAA would not have asked for this review absent a strong interest in improving its mod- eling capabilities. It is in this context that the committee wishes to express its strong view that the current model is deﬁcient for estimating PTT and that continuing to iterate on it in the same manner as in the past while not incorporating more complete and representative task performance data will do little to improve this situation. Looking farther out, the durability of the task load model for PTT analysis and for other possible applications, such as to inform traffic flow planning, will depend not only on the successful gathering and use of task performance data but also on the nature and pace of change in the air traffic control enterprise. Developments anticipated for the planned Next Generation Air Transportation System (NextGen), such as increased automation and many more decision-support tools, could substantially alter controller roles and responsibilities in ways that are highly relevant to the modeling of PTT. Without more knowledge about the nature and timing of these NextGen changes, it is not possi- ble to predict how the model will hold up structurally, much less how
OCR for page 58
Findings and Recommendations 63 changes in trafﬁc data, task coverage, and task times might make it more adaptable. RECOMMENDATIONS In commencing its review, the committee expected to find—but did not—strong documentation explaining the logic and structure of the model and evidence of its having been the subject of statistical tests and other scientiﬁc methods for establishing and validating model parame- ters, assumptions, and output. More rigorous documentation and peer review during earlier stages of model development would likely have exposed many of the problems identiﬁed in this report, providing earlier opportunities to avoid or correct them. Nevertheless, as preface to offer- ing advice on ways to improve the modeling process going forward, it is important to restate the ﬁnding that the current model framework, despite the data shortcomings, represents a major improvement over past modeling methods to inform workforce planning. In the following recommendations it is presumed that FAA will elect to retain the core model and invest meaningfully in its improvement. Observe and Measure Controller Task Performance Through more systematic and carefully designed observation and analysis of controller performance, model developers should gain a better under- standing of the tasks that controllers perform in working en route trafﬁc, how they perform them, and the time required to do so. The gathering and analysis of data on controllers working alone and interacting in teams, whether through ﬁeld observations or HITL experiments, should be the primary method to identify and elicit information on controller tasks. Model All Controller Tasks Modeling all tasks that contribute signiﬁcantly to total controller task load is fundamental for estimating PTT. FAA should use the informa- tion gained from observing, measuring, and analyzing controller task performance to quantify the task load associated with the services pro- vided by both the lead and associate controllers. The modeling of all
OCR for page 58
64 Air Trafﬁc Controller Stafﬁng in the En Route Domain controller tasks will eliminate the need to infer task load to derive esti- mates of PTT. Using a single model for estimating task load rather than separate ones for each controller is the preferred approach, since it will facilitate both PTT conversion and model validation. Validate Model Elements Task performance data should be used also to assess the validity and impact of all key modeling processes, relationships, and assumptions. Because it is not possible to validate PTT estimates against actual stafﬁng levels, ensuring that the model elements are well justiﬁed and viewed as credible is vitally important. Examples of modeling assumptions that would seem to warrant early attention are those that concern task per- formance by the controllers when working alone and in teams, whether tasks are performed sequentially or concurrently, and how total task load affects the pace of task performance.