TRB Special Report 301: Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration's Task Load Model (2010)

The Federal Aviation Administration (FAA) has supported the development of a quantitative model that estimates the task load on controllers created by air traffic activity in each of the more than 750 sectors of the nation’s en route airspace. The model uses traffic operations and flight-planning data to simulate the traffic activity in each sector. It then associates with this traffic the specific controller tasks that must be performed, computes and assigns a time to perform each task, and calculates the total time spent by controllers on the modeled tasks. FAA has been using the model’s task load output to estimate the number of controllers required to work the traffic in each sector, an estimate known as “positions to traffic” (PTT).

FAA asked the National Academies to convene an expert committee to examine and offer advice where appropriate for improving (a) the overall technical approach of task-based modeling, (b) input data and processes used for modeling traffic activity, (c) tasks and methods used to assign task times, and (d) means for validating model assumptions, parameters, and output. In addressing this charge, the committee was asked to be cognizant of the “overall tradeoffs made due to data availability” and to consider the “adaptability of the approach to reflect changes in the tasks of controllers as their roles evolve over time.”

Findings from the assessments in the previous chapters are provided next, including those relevant to the use of task load output for estimating PTT. These findings are followed by recommendations for model improvements.

The results of task-based modeling can be a valuable source of objective information for workforce planning, and FAA’s current model is a marked improvement over previous models that did not account for traffic complexity. The basic structure of the CAASD model, in which traffic activity is simulated and controller tasks and task times are associated with traffic, represents a logical approach to estimating task load.

Compared with simple traffic counts, the simulations of traffic in the CAASD model provide a more complete picture of both the volume and nature of traffic activity in the en route domain. The traffic activity is modeled in sufficient depth and resolution to enable reasonable approximations of traffic complexity and associated controller tasks. CAASD can check the model results against records of actual traffic activity to improve the traffic modeling capabilities.

The nine tasks in the model appear to be representative of the main R-side services that must be performed to work traffic—although whether the specific claim that 90 percent of R-side tasks are covered has not been well established. Compared with the other eight modeled tasks, the monitoring task is treated in the most confusing manner and is difficult to connect with traffic activity. A simpler and more transparent means of estimating monitoring time deserves consideration. While the model’s coverage of R-side tasks may be sufficient for traffic capacity analysis, the omission of all tasks performed by the associate controller makes its task load output inadequate for estimating PTT.

Many of the task times in the model are derived from a separate modeling process known as Goals, Operators, Methods, and Selection Rules (GOMS). For seven of the nine modeled tasks, GOMS is used to derive

task times. The other task times are developed through consultations with subject matter experts. None of the task times is derived from the observation and analysis of controllers performing tasks in the field or in experiments. The GOMS-derived times are based largely on expert judgment and are only loosely validated against a limited set of task performance data obtained from human-in-the-loop (HITL) experiments conducted for other purposes. The use of GOMS to derive many task times, coupled with reliance on expert judgment for validating these modeled times and for estimating many others, raises serious questions about the accuracy of the model’s task load output.

Summing all of the time spent on tasks may be the most practical approach for computing total task load. However, adding one task time to another does not account for the possibility—and real-world probability—that some tasks are performed concurrently. The additive approach also does not account for the possibility that the time it takes to perform a specific task may vary depending on the level of traffic activity and the number of controllers working the sector. Taking these possibilities into account may not have meaningful effects on the modeled task load. Examining their potential effects, however, is important for making this case.

FAA and model developers have sought to compensate for the absence of D-side task load by employing various processes that infer total task load to facilitate conversion to PTT. All of the PTT conversion methods used, including the current method of fuzzy logic modeling, exhibit the same fundamental flaw—they imply an understanding of total task load without ever identifying the unmodeled tasks, much less measuring the time it takes to perform them. The conversions rely on experts to determine thresholds and to assign complexity weightings to the unidentified and unmodeled tasks. The D-side task loads implied by these thresholds and weightings are not validated, nor can they be in the absence of any empirical data on task performance. On the whole, the use of the fuzzy logic modeling to infer task load adds little more than spurious precision

to the PTT estimates while complicating and reducing the transparency of the modeling process.

Modeled traffic activity can be checked for accuracy through comparisons with records of actual traffic. In contrast, validating PTT estimates is more challenging since there is no external measure of staffing requirements against which the accuracy of the estimates can be judged. Analyzing staffing records is of limited value since the main purpose of PTT modeling is to find out when staffing levels can be better aligned with traffic demand. The main method by which model developers have sought to assess PTT estimates is by presenting them to groups of experts, often consisting of individuals who manage and staff the en route centers. Yet such checks can suffer from the same shortcoming that limits the value of comparisons with staffing records—the potential for bias toward existing staffing practice.

Because PTT estimates cannot be assessed through direct observation, all of the model’s key assumptions, processes, and parameters must be well justified and validated. A lack of data on task performance precludes validation of the task times constructed from GOMS and the task complexity weightings used in the fuzzy logic conversion method. The deficiencies of these two modeling processes go well beyond parameter validation, as explained earlier. Yet the lack of empirical data on task performance has hindered validation throughout the modeling process, from assessing key assumptions about tasks being performed sequentially and at a fixed pace to characterizing the tasks handled by the associate controller.

In the study charge, FAA asked the committee to be cognizant of tradeoffs that must be made because of limited data availability, which presumably refers to the cost and complications of obtaining task performance data. FAA also asked for advice on the model’s adaptability to reflect changes in controller roles and tasks over time.

Many of the findings cited earlier point to a need for a firmer empirical basis both for evaluating the structure of the model and for estimating

the values of the parameters used in it. By and large, the model was developed and has been evaluated with heavy reliance on the insights and opinions obtained from subject matter experts and facility personnel. More objective and quantitative task performance data are clearly needed, not only for developing the model parameters and evaluating the task load output but also for including more controller tasks in the modeling of PTT. The committee recognizes that gathering such data from operational and experimental settings will require more resources and access to controllers, which may present budget and labor relations issues. Although such cost implications were not examined in this study, it must be pointed out that there is a cost in model credibility from not obtaining such data. This cost is manifested in many ways throughout the current model, from the added opaqueness caused by fuzzy logic modeling to the excessive reliance on expert opinion and judgment for model development and validation.

Whether FAA is committed to taking this data-gathering step will presumably depend on its assessment of the cost trade-offs and its plans for using the model for a long time and for other possible purposes. Not knowing these plans, the committee nevertheless believes FAA would not have asked for this review absent a strong interest in improving its modeling capabilities. It is in this context that the committee wishes to express its strong view that the current model is deficient for estimating PTT and that continuing to iterate on it in the same manner as in the past while not incorporating more complete and representative task performance data will do little to improve this situation.

Looking farther out, the durability of the task load model for PTT analysis and for other possible applications, such as to inform traffic flow planning, will depend not only on the successful gathering and use of task performance data but also on the nature and pace of change in the air traffic control enterprise. Developments anticipated for the planned Next Generation Air Transportation System (NextGen), such as increased automation and many more decision-support tools, could substantially alter controller roles and responsibilities in ways that are highly relevant to the modeling of PTT. Without more knowledge about the nature and timing of these NextGen changes, it is not possible to predict how the model will hold up structurally, much less how

changes in traffic data, task coverage, and task times might make it more adaptable.

In commencing its review, the committee expected to find—but did not—strong documentation explaining the logic and structure of the model and evidence of its having been the subject of statistical tests and other scientific methods for establishing and validating model parameters, assumptions, and output. More rigorous documentation and peer review during earlier stages of model development would likely have exposed many of the problems identified in this report, providing earlier opportunities to avoid or correct them. Nevertheless, as preface to offering advice on ways to improve the modeling process going forward, it is important to restate the finding that the current model framework, despite the data shortcomings, represents a major improvement over past modeling methods to inform workforce planning. In the following recommendations it is presumed that FAA will elect to retain the core model and invest meaningfully in its improvement.

Through more systematic and carefully designed observation and analysis of controller performance, model developers should gain a better understanding of the tasks that controllers perform in working en route traffic, how they perform them, and the time required to do so. The gathering and analysis of data on controllers working alone and interacting in teams, whether through field observations or HITL experiments, should be the primary method to identify and elicit information on controller tasks.

Modeling all tasks that contribute significantly to total controller task load is fundamental for estimating PTT. FAA should use the information gained from observing, measuring, and analyzing controller task performance to quantify the task load associated with the services provided by both the lead and associate controllers. The modeling of all

controller tasks will eliminate the need to infer task load to derive estimates of PTT. Using a single model for estimating task load rather than separate ones for each controller is the preferred approach, since it will facilitate both PTT conversion and model validation.

Task performance data should be used also to assess the validity and impact of all key modeling processes, relationships, and assumptions. Because it is not possible to validate PTT estimates against actual staffing levels, ensuring that the model elements are well justified and viewed as credible is vitally important. Examples of modeling assumptions that would seem to warrant early attention are those that concern task performance by the controllers when working alone and in teams, whether tasks are performed sequentially or concurrently, and how total task load affects the pace of task performance.