Summary

The Federal Aviation Administration (FAA) is seeking to improve a mathematical model that estimates the time spent by controllers performing tasks in working the air traffic in each of the more than 750 sectors of the nation’s en route airspace. FAA has been using the model’s estimates of task time expenditure, or “task load,” to assess the number of controllers required to work each sector’s traffic. The model simulates the traffic activity experienced in each sector and then associates task times with this activity to compute task load. While the task load values do not portray the total workload on controllers—since workload is driven by other factors such as stress, fatigue, and expertise—they can provide a consistent and objective source of information for controller staffing. It is for this reason that an earlier TRB report1 urged FAA to pursue task-based modeling for workforce planning.

FAA’s task load model is currently being used as one of several inputs in the agency’s annual controller workforce plan (CWP). The modeled task loads are used to estimate the number of controllers required in position in each sector to perform the traffic-driven tasks, which FAA refers to as “positions to traffic,” or PTT. When a sector is open to traffic, it has at least one controller in position, the lead controller. Depending on traffic demand and other factors, the lead controller may be accompanied by an associate controller. Thus, PTT values are usually 1 or 2. When traffic is exceptionally heavy a third controller may be added to the team, although this setup is seldom a planned staffing configuration.

Having used the PTT estimates from the model to inform the CWP for the past few years, FAA sought an independent review of the modeling

1

TRB. 1997. Special Report 250: Air Traffic Control Facilities—Improving Methods to Determine Staffing Requirements. National Research Council, Washington, D.C.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary The Federal Aviation Administration (FAA) is seeking to improve a mathematical model that estimates the time spent by controllers per- forming tasks in working the air traffic in each of the more than 750 sec- tors of the nation’s en route airspace. FAA has been using the model’s estimates of task time expenditure, or “task load,” to assess the number of controllers required to work each sector’s traffic. The model simulates the traffic activity experienced in each sector and then associates task times with this activity to compute task load. While the task load values do not portray the total workload on controllers—since workload is driven by other factors such as stress, fatigue, and expertise—they can provide a consistent and objective source of information for controller staffing. It is for this reason that an earlier TRB report1 urged FAA to pur- sue task-based modeling for workforce planning. FAA’s task load model is currently being used as one of several inputs in the agency’s annual controller workforce plan (CWP). The modeled task loads are used to estimate the number of controllers required in posi- tion in each sector to perform the traffic-driven tasks, which FAA refers to as “positions to traffic,” or PTT. When a sector is open to traffic, it has at least one controller in position, the lead controller. Depending on traf- fic demand and other factors, the lead controller may be accompanied by an associate controller. Thus, PTT values are usually 1 or 2. When traffic is exceptionally heavy a third controller may be added to the team, although this setup is seldom a planned staffing configuration. Having used the PTT estimates from the model to inform the CWP for the past few years, FAA sought an independent review of the modeling 1 TRB. 1997. Special Report 250: Air Traffic Control Facilities—Improving Methods to Determine Staffing Requirements. National Research Council, Washington, D.C. 1

OCR for page 1
2 Air Traffic Controller Staffing in the En Route Domain process to assess its utility and validity going forward. Specifically, FAA asked the National Academies to convene an expert committee to exam- ine and offer advice where appropriate for improving (a) the overall technical approach of task-based modeling, (b) input data and processes used for modeling traffic activity, (c) tasks and methods used to assign task times, and (d) means for validating model assumptions, parameters, and output. In addressing this charge, the committee was asked to be cognizant of the “overall tradeoffs made due to data availability” and to consider the “adaptability of the approach to reflect changes in the tasks of controllers as their roles evolve over time.” Key study findings with respect to each of these elements of the study charge are given next, fol- lowed by recommendations. KEY FINDINGS Task-Based Approach The results of task-based modeling can be a valuable source of objective information for workforce planning, and FAA’s current model is a marked improvement over previous models. Earlier models measured the num- ber of aircraft flying through a sector without accounting for the vari- ability in the complexity of this traffic, and thus the variability in controller tasks and time demands. For example, aircraft changing headings and altitude create more traffic complexity than aircraft cruising straight through a sector. FAA’s current model accounts for traffic complexity by simulating the traffic flows and patterns experienced in the en route sec- tors and relating them to the time-varying tasks that controllers perform. The basic model structure, in which traffic activity is simulated and con- troller tasks and task times are associated with traffic, represents a logi- cal approach to estimating task load. The methods used to derive model parameters and values and to convert the modeled task load into PTT are the subject of most of the criticism and advice in this report. Simulations of Traffic Activity By using available traffic operations and flight-planning data, flight plans, and trajectory modeling, the task load model simulates past sec-

OCR for page 1
Summary 3 tor traffic flows and patterns. The traffic activity is modeled in sufficient depth and resolution to enable reasonable approximations of traffic complexity and associated controller tasks. Because the simulated traf- fic can be checked against records of actual traffic activity, there is ample opportunity to use empirical data to validate output accuracy and guide the development and calibration of the traffic modeling methods and parameters. Model developers have been taking advantage of these opportunities to make periodic improvements to the traffic modeling process. Task Coverage The task load model does not analyze all of the tasks performed by con- trollers but only certain ones performed by the lead controller in com- municating with aircraft, monitoring flights on the radar screen, and communicating with controllers from other sectors and centers. The modeling of these lead controller tasks is essential for analyzing the traf- fic throughput capacity of individual sectors, which was the original purpose of the model. Task coverage for this purpose appears to be ade- quate. Yet in order to know when the demands of traffic necessitate more than one controller—that is, in order to estimate PTT—it is necessary to know the total task load on controllers, including the task load on the associate controller. By omitting all of the tasks performed by the asso- ciate controller, the model’s task load output alone is not adequate for estimating PTT. FAA and model developers have sought to compensate for this signif- icant gap in task coverage by employing various processes that infer the missing task load to enable conversions of model output into PTT. All of the PTT conversion methods used, including the current one using fuzzy logic modeling, exhibit the same fundamental flaw—they imply an abil- ity to estimate total task load without ever identifying the unmodeled tasks, much less measuring the time it takes to perform them. The PTT conversions using fuzzy logic modeling rely on experts to assign com- plexity weightings to the unidentified and unmodeled tasks. These weightings are not validated, nor can they be in the absence of any empir- ical data on task performance. On the whole, the use of fuzzy logic mod- eling to infer task load adds little more than spurious precision to the

OCR for page 1
4 Air Traffic Controller Staffing in the En Route Domain PTT estimates while complicating and reducing the transparency of the modeling process. Derivation of Task Times Since task load output is the sum of the time spent by controllers per- forming tasks, the task completion times are critical model parameters. Many of the task times in the model are derived from a separate model- ing process known as Goals, Operators, Methods, and Selection Rules (GOMS). The GOMS-derived times are based largely on expert judgment and are only loosely validated against a limited set of task performance data obtained from human-in-the-loop (HITL) experiments conducted for other purposes. GOMS modeling is typically used where conditions do not permit the observation and analysis of task performance in oper- ational or experimental settings. The committee believes that such con- ditions do not exist in the air traffic control domain to the extent that warrants such heavy reliance on GOMS. The use of GOMS to derive many task times, coupled with reliance on expert judgment for validating these modeled times and for estimating many others, raises serious ques- tions about the accuracy of the model’s task load values. Validation Modeled traffic activity can be checked for accuracy through compar- isons with records of actual traffic. In contrast, validating PTT estimates is more challenging since there is no external measure of staffing require- ments against which the accuracy of the estimates can be judged. Ana- lyzing staffing records is of limited value since the main purpose of PTT modeling is to find out when staffing levels can be better aligned with traffic demand. A main means by which model developers have sought to assess PTT estimates is by presenting them to groups of experts, often consisting of individuals who manage and staff the en route centers. Yet such checks can suffer from the same shortcoming that limits the value of comparisons with staffing records—the potential for bias toward exist- ing staffing practice. Because PTT estimates cannot be assessed through direct observation, all of the model’s key assumptions, processes, and parameters must be well

OCR for page 1
Summary 5 justified and validated. A lack of data on task performance precludes vali- dation of the task times constructed from GOMS and the task complexity weightings used in the fuzzy logic conversion method. The deficiencies of these two modeling processes go well beyond parameter validation, as explained earlier. Yet the lack of empirical data on task performance has hindered validation throughout the modeling process, from assessing key assumptions about tasks being performed sequentially and at a fixed pace to characterizing the tasks handled by the associate controller. Data Availability and Model Adaptability In the study charge, FAA asked the committee to be cognizant of trade- offs that must be made because of limited data availability, which presum- ably refers to the cost and complications of obtaining task performance data. FAA also asked for advice on the model’s adaptability to reflect changes in controller roles and tasks over time. Many of the findings cited earlier point to a need for a firmer empir- ical basis both for evaluating the structure of the model and for estimat- ing the values of the parameters used in it. By and large, the model was developed and has been evaluated with heavy reliance on the insights and opinions obtained from subject matter experts and facility person- nel. More objective and quantitative task performance data are clearly needed, not only for developing the model parameters and evaluating the task load output but also for including more controller tasks for the modeling of PTT. The committee recognizes that gathering such data from operational and experimental settings will require more resources and access to controllers, which may present budget and labor relations issues. Although such cost implications were not examined in this study, it must be pointed out that there is a cost in model credibility from not obtaining such data. This cost is manifested in many ways throughout the current model, from the added opaqueness caused by fuzzy logic modeling to the excessive reliance on expert opinion and judgment for model development and validation. Whether FAA is committed to taking this data-gathering step will pre- sumably depend on its assessment of the cost trade-offs and its plans for using the model for a long time and for other possible purposes. Not knowing these plans, the committee nevertheless believes that FAA would

OCR for page 1
6 Air Traffic Controller Staffing in the En Route Domain not have asked for this review absent a strong interest in improving its modeling capabilities. It is in this context that the committee wishes to express its strong view that the current model falls short in its ability to estimate PTT and that continuing to iterate on it in the same manner as in the past while not incorporating more complete and representative task performance data will not overcome the deficiency. Looking farther out, the durability of the task load model for PTT analysis and for other possible applications, such as to inform traffic flow planning, will depend not only on the successful gathering and use of task performance data but also on the nature and pace of change in the air traffic control enterprise. Developments anticipated for the planned Next Generation Air Transportation System (NextGen), such as increased automation and many more decision-support tools, could substantially alter controller roles and responsibilities in ways that are highly relevant to the modeling of PTT. Without more knowledge about the nature and timing of these NextGen changes, it is not possi- ble to predict how the model will hold up structurally, much less how changes in traffic data, task coverage, and task times might make it more adaptable. RECOMMENDATIONS In commencing its review, the committee expected to find—but did not—strong documentation explaining the logic and structure of the model and evidence of its having been the subject of statistical tests and other scientific methods for establishing and validating model parame- ters, assumptions, and output. More rigorous documentation and peer review during earlier stages of model development would likely have exposed many of the problems identified in this report, providing earlier opportunities to avoid or correct them. Nevertheless, as a preface to offering advice on ways to improve the modeling process going forward, it is important to restate the finding that the current model framework, despite the data shortcomings, represents a major improvement over past modeling methods to inform workforce planning. In the following recommendations it is presumed that FAA will elect to retain the core model and invest meaningfully in its improvement.

OCR for page 1
Summary 7 Observe and Measure Controller Task Performance Through more systematic and carefully designed observation and analy- sis of controller performance, model developers should gain a better understanding of the tasks that controllers perform in working en route traffic, how they perform them, and the time required to do so. The gath- ering and analysis of data on controllers working alone and interact- ing in teams, whether through field observations or HITL experiments, should be the primary method to identify and elicit information on con- troller tasks. Model All Controller Tasks Modeling all tasks that contribute significantly to total controller task load is fundamental for estimating PTT. FAA should use the informa- tion gained from observing, measuring, and analyzing controller task performance to quantify the task load associated with the services pro- vided by both the lead and associate controllers. The modeling of all con- troller tasks will eliminate the need to infer task load to derive estimates of PTT. Using a single model for estimating task load rather than sepa- rate ones for each controller is the preferred approach, since it will facil- itate both PTT conversion and model validation. Validate Model Elements Task performance data should be used also to assess the validity and impact of all key modeling processes, relationships, and assumptions. Because it is not possible to validate PTT estimates against actual staffing levels, ensuring that the model elements are well justified and viewed as credible is vitally important. Examples of modeling assumptions that would seem to warrant early attention are those that concern task per- formance by the controllers when working alone and in teams, whether tasks are performed sequentially or concurrently, and how total task load affects the pace of task performance.

OCR for page 1