Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 46
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model 4 Converting Task Load into Positions to Traffic Converting task load estimates into positions to traffic (PTT) requires knowing when the total time spent by the lead controller on both R- and D-side tasks reaches a point where assistance from an associate controller is needed. As explained in the previous chapter, the CAASD model produces estimates of the R-side task load only. The challenge, therefore, is in finding ways to convert this limited task load into values for PTT. The first part of this chapter describes the methods employed by CAASD to make these conversions. The second part consists of CAASD’s own evaluations of the conversion results, and the last part gives the committee’s assessment of the methods. CONVERSION METHODS Time Threshold CAASD has used two basic methods for converting the modeled R-side task load into PTT. The first, which is no longer being used, presumes that once R-side task load reaches a given threshold, then an associate (D-side) controller is needed to help work the traffic in the sector. The time threshold originally used by modelers—600 seconds during a 900-second (15-minute) period—presumes that at this point the combination of modeled R-side tasks and unmodeled D-side tasks fully occupies the controlling time available to the lead controller. The specific 600-second threshold was identified by model developers on the basis of consultations with facility managers, who found that the resulting PTT values were closer to their expectations of staffing levels associated with the modeled traffic than those generated by other cutoff points (550, 580, etc.). Figure 4-1
OCR for page 47
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model FIGURE 4-1 Example use of 600-second (R-side) task load threshold to estimate PTT over 8-hour period.
OCR for page 48
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model provides a graphic representation of the R-side task load converted to PTT in this manner for a single sector for an 8-hour block of time. Nevertheless some of the facility managers questioned whether a single threshold was appropriate for predicting PTT across sectors that encountered wide variability in traffic patterns and complexity. They observed, for instance, that the D-side task load tends to be higher for some types of traffic activity than for others, which would imply the need for a second controller at a threshold lower than 600 seconds for R-side task loads generated by such traffic. For instance, in some sectors the complexity of traffic may be relatively straightforward, consisting of mostly entries and exits as aircraft transit a sector, which generates minimal D-side work. In other sectors (or even in the same sector at a different time of day), traffic patterns may be more complex, consisting of more delays, international entries and exits, and nonradar arrivals and departures, which creates much more D-side work. In response to these concerns, CAASD added new triggers to its model for R-side tasks associated with international and nonradar traffic activity. In addition, new rules for nonradar and international tasks were added to adjust the 600-second conversion. Specifically, if the total R-side task load were less than 600 seconds but consisted of at least 100 seconds of time spent on nonradar tasks or if more than five aircraft were in the sector when any amount of time was spent on nonradar tasks, then two controllers were assumed to be needed. Likewise, if total R-side task load were less than 600 seconds, but international task load exceeded 40 seconds or if more than 10 aircraft were in the sector along with any amount of time spent on international traffic, then two controllers were assumed to be needed. CAASD referred to this rule-adjusted conversion as the “600-plus” method. In general, the PTT estimates from this adjusted method were found to be more in line with the expectations of operational personnel consulted from facilities having significant international and nonradar activity. Fuzzy Logic Modeling Even after rules for international and nonradar tasks were applied, CAASD worried that other variability in sector traffic patterns was creating even more variability in total task load than could be accounted for by the
OCR for page 49
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model modified 600-second cutoff. Accordingly, CAASD considered developing even more rule sets, basically extending its “600-plus” method. To do so, however, required more information on the D-side task load associated with different patterns and volumes of traffic activity. Yet CAASD only had the qualitative judgments of operational experts to assess total task load—that is, their judgments about whether one traffic profile, or R-side task load, tended to create more or less D-side task load. CAASD concluded that the inference rules used in fuzzy logic modeling might be well suited to inferring total task load from these qualitative judgments. In an explanatory document submitted to the committee,1 CAASD described the fuzzy modeling process and its purpose as follows: Fuzzy logic involves setting multiple thresholds for each input variable, and then creating rules of interaction. The technique has three distinct steps: fuzzification, inference, and defuzzification. Each of these steps is discussed below relative to the fuzzy model for PTT. Fuzzification is the process by which a degree of membership is determined for each of the eight task workload inputs (entry, monitor, exit, transition, separation, delay, international, and nonradar). Three overlapping fuzzy terms were used for all task workloads: “low,” “medium,” and “high.” These terms are referred to as linguistic variables and represent the relative degrees of difficulty, i.e., total team workload for increasing degrees of specific R-controller task workload. The membership function for each of the three terms, which ranges from 0 to 1, was calibrated separately for each workload task. For each of the tasks, there is an inflection point where membership is equal to 1. Figure 4-2 shows an example of how the Entry input variable looks in the software interface. The second step in fuzzy modeling is to apply inference rules. Once the level of membership has been determined for each task workload relative to each linguistic variable (i.e., fuzzification), this level of membership is combined with similar information for other task workload in the same grouping. Three task groups are used in the model: basic tasks, complex tasks, and other tasks. These task groups were chosen based on their staffing impact. Basic workload tasks alone (entry, exit and monitor), will not require a second or third controller, unless they are relatively elevated due to high traffic levels. However, if there is a significant amount of complex task workload present (spacing, delay, and transition), an R-side will be more likely to need 1 The committee asked CAASD to draft a paper explaining the task model load and processes used to convert its output into PTT. The quoted sections that follow are derived from this submitted paper, which can be obtained from TRB.
OCR for page 50
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model FIGURE 4-2 Entry input variable in fuzzy logic model software interface. assistance. Also, as mentioned earlier, if other task workload (international and nonradar) is present, then it is highly likely that a D-side is present to assist with the nonautomated hand-offs. Each of the three task groupings uses a system of “if–then” statements to translate individual task weights into task group weights, and then a final task group weighting is translated into an estimate of the number of controllers needed. Varying degrees of these rules are simultaneously activated or “fired” based on the individual contributions of the input linguistic variables (e.g., a medium will contribute more than a low). Figure 4-3 illustrates the framework used in the PTT fuzzy model. The final step in the process is defuzzification. It involves applying all of the inference rules, which are weighted, to obtain a definitive solution value. The model produces both a discrete number (0, 1, 2, or 3) and a value that can be a fractional value between 0 and 3. The discrete value is generated by a process known as the Mean of Maximum (MoM) method and is the method used to translate the final degree of membership into the discrete PTT variable (0, 1, 2 or 3). This methodology is typically used for classification problems and produces the most plausible or likely result. The other method that produces the fractional value between 0 and 3 is known as Center of Maximum (CoM). Although the fractional value is not used for the PTT data provided to the FAA for the CWP staffing models, it has been useful in calibration of the model. It indicates whether the model was close to producing a different value for the discrete method used for the PTT data. For example, a value of 2.4 indicates that the workload is trending towards the need of a third (T-side) controller. Essentially, the translation produced by the PTT fuzzy model reflects how operational experts characterize position needs: low degree of workload is equivalent to one controller, medium degree of workload is equivalent to
OCR for page 51
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model FIGURE 4-3 Framework for fuzzy logic modeling process to infer PTT.
OCR for page 52
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model two controllers, and high degree of workload is equivalent to three controllers. The translation is performed without having to define “low,” “medium,” and “high.” The last paragraph in CAASD’s description explains what the fuzzy logic model is doing to generate PTT values. In essence, it is assigning additional task time to each of the modeled R-side tasks based on expert opinions on the associated D-side task load. However, the implied D-side tasks are never identified, nor are the times assigned to them by the experts made explicit. Nevertheless, the D-side task loads must be determined in order to generate the PTT values associated with the various combinations of modeled R-side task load. That such D-side task load estimates are being made, however vaguely and opaquely, raises questions about the validity of this modeling process and whether the characterizations of the operational experts are indeed accurate. The committee sought, but was not was presented with, the total task loads that are implied by the different combinations of R-side tasks that generate specific PTT values. Making explicit these implied D-side task loads so that they can be assessed is essential to judging the validity of the PTT estimates produced through fuzzy logic modeling. Although not checked in this most fundamental manner, the fuzzy logic modeling process is being used by CAASD for its PTT conversions, and model developers have indicated satisfaction with the results. In the next section, the methods used by model developers to assess the conversions are examined. CAASD EVALUATIONS OF PTT CONVERSIONS The PTT conversions were evaluated by CAASD primarily on the basis of (a) consultations with operational experts asking them to judge whether the results seem reasonable and (b) comparisons of the model output with available operational data and staffing records. A dilemma in consulting facility personnel and staffing records to validate PTT estimates is that a central purpose of PTT modeling is to assess whether staffing levels are aligned with experienced traffic demand. A problem with asking facility personnel to assess PTT estimates is that their response may be skewed by a view that existing staffing levels are
OCR for page 53
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model optimal. Likewise, consulting staffing records can be problematic. On the one hand, if the records show that one controller successfully worked the traffic in a sector when the model estimated a need for two controllers, then such a comparison could be helpful in identifying modeling problems that tend to overestimate PTT. Indeed, it is through such comparisons that volume-based methods of generating PTT were found to be lacking. On the other hand, if the PTT model indicates a need for one controller when staffing records show that two were in position, it is much more difficult to ascertain whether the model underestimated the need for a second controller or whether actual staffing levels were too high for the experienced traffic activity. Review of PTT Estimates by Facility Personnel To assess its PTT estimates, CAASD presented the results to managers and controllers at 13 centers spread across the country.2 At each evaluation session, participants were given a general overview of the modeling process. Controllers then analyzed the task load and PTT output for a high-traffic day as well as for overall seasonal staffing averages for their area. Specific feedback was sought on the accuracy of the model in capturing the type and quantity of task load as well as in producing reasonable PTT estimates. The results of these evaluations were apparently used to modify the task load model and the conversion methods, although the specific adjustments made in response to each facility visit were not explained to the committee. Nevertheless, according to CAASD, the recommended number of changes to the model declined with each center visit. CAASD took these developments as indicative of a model that was providing an increasingly accurate portrayal of PTT. Comparison with Staffing Records In addition to these center visits, model developers examined sector staffing records as a point of reference for evaluating the PTT estimates. FAA’s controller time and attendance system, known as Cru-ART, contains 2 Albuquerque, New Mexico; Boston, Massachusetts; Chicago, Illinois; Indianapolis, Indiana; Jacksonville, Florida; Kansas City, Missouri; Memphis, Tennessee; Miami, Florida; Minneapolis, Minnesota; New York City; Oakland, California; Salt Lake City, Utah; and Washington, D.C.
OCR for page 54
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model reports of the number of controllers who timed into a sector during a given time period. CAASD noted to the committee, however, that the Cru-ART data do not always show the number of controllers actually required to work the traffic because some controllers are timed in for training purposes. CAASD therefore looked for opportunities to use the Cru-ART data3 in ways that would minimize the influence of some of its shortcomings. Table 4-1 shows the results of a Cru-ART analysis presented to the committee. In this case, CAASD compared the number of controllers recorded on position with the number that would have been estimated using the earlier traffic-volume method and using the task load model’s output converted to PTT using the 600-second and fuzzy logic methods. Traffic operation counts were evaluated for each of the nation’s 20 en route centers to identify the 90th-percentile traffic days, that is, those days in which traffic volumes were higher than those experienced in 90 percent of the other days during the year. In particular, two days close to the 90th percentile for each center were chosen for the analysis, focusing on the staffing shifts between 7:00 a.m. and 11:00 p.m.4 Using the Cru-ART records, CAASD calculated the total number of controllers working the traffic during these periods for each center. These totals were then compared with the PTT estimates from the models and task load conversion methods cited earlier. The results from this analysis show that the previous volume-based method of PTT estimation, which does not consider the complexity of the traffic, consistently overstated the number of controllers required to work the traffic when compared with the Cru-ART records of actual staffing levels. Indeed, in most of the centers, the volume-based methods led to estimates of PTT that were 17 to 70 percent higher than the Cru-ART numbers. By comparison, the 600-second threshold yielded results much closer to the staffing levels indicated by Cru-ART, although the values were mostly lower. The fuzzy logic method came closest to the staffing levels indicated by the Cru-ART records. Because the fuzzy logic model tries to take into account differences in sector traffic complexity and the 3 The Cru-ART data were processed to isolate the “PTT-like” information for each en route center. 4 Staffing for the midnight shift is often based on factors in addition to PTT needs; thus only 7:00 a.m. to 11:00 p.m. local time was used for the comparative analysis.
OCR for page 55
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model TABLE 4-1 Percent Difference in PTT Estimates: Earlier Volume-Based Method for Assessing Staffing Compared with Results from Task Load Model Using Two Alternative Conversion Methods ARTCC Traffic Volume-Based Methoda CAASD Task Load Model Results Using 600-second Conversionb CAASD Task Load Model Results Using Fuzzy Logic Conversion ZAB 70 2 7 ZAU 27 −14 −1 ZBW 49 2 10 ZDC 70 17 26 ZDV 68 5 10 ZFW 34 −14 −6 ZHU 34 −15 −5 ZID 36 −6 3 ZJX 47 −10 −1 ZKC 50 −5 0 ZLA 40 −5 7 ZLC 54 −10 −6 ZMA 29 −15 −5 ZME 41 −11 −1 ZMP 34 −18 −8 ZNY 41 −6 9 ZOA 17 −21 −11 ZOB 32 −8 1 ZSE 19 −21 −10 ZTL 38 −10 1 aThe volume-based method uses simple traffic counts in sectors as the basis for calculating controller workload. bCAASD did not assess the “600-second plus” conversion method. NOTE: The numbers in the table indicate the percentage difference in positions estimated by each model and conversion method when compared with historical Cru-ART staffing records for the same time period (e.g., a value of 50 means that the model and its conversion method led to a PTT estimate that is 50 percent higher than the number of positions indicated by Cru-ART recorded staffing levels). resultant impacts on total controller task load, CAASD believes that this is why the latter conversion method yields PTT values that are closer to the Cru-ART numbers. It merits reiterating, however, that comparing PTT estimates with staffing records is problematic because these records are not necessarily indicative of the staffing that was required to work the experienced traffic. Thus, it not possible to conclude on the basis of this particular analysis
OCR for page 56
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model that the conversion method using fuzzy logic modeling yields any more accurate predictions of PTT than those yielded by the 600-second conversion method. Indeed, if actual staffing levels (as indicated by Cru-ART) were much too high, the lower PTT values generated by the 600-second conversion may have been more reflective of actual staffing needs. To be sure, however, the analysis in Table 4-1 brings into question the accuracy of the simple volume-based method for estimating PTT. The PTT values produced through this method are much higher than the staffing numbers in Cru-ART. If one assumes that the controllers staffing the sectors were able to meet their traffic management responsibilities, then clearly these volume-based staffing levels were far too high. COMMITTEE ASSESSMENT The CAASD task load model examines only one set of controller tasks: the R-side tasks performed by the lead controller. Because of this limitation, use of the model results to estimate PTT requires either supplemental measures of D-side task load or a creative means of converting the model output into measures of total task load. CAASD decided against obtaining information on D-side task load. Instead, model developers pursued an alternative approach that has relied heavily on expert judgment rather than data. The following recap of this process makes very clear the weakness of this approach. In its initial efforts to convert the task load output to PTT, model developers consulted with operational experts to estimate when total R-side task load equated to an accompanying amount of D-side task load to warrant a second controller. These consultations, which did not involve any measurement of D-side tasks, apparently led CAASD to conclude that 600 seconds of R-side task load during a 15-minute period was the appropriate threshold. To validate this expert-derived threshold, CAASD again consulted with operational experts to assess the PTT estimates that resulted. The advice from these experts caused CAASD to make further adjustments to the threshold to account for the additional D-side work that accompanies certain kinds of traffic, such as international and nonradar operations. These iterations produced results closer to the expectations of consulted facility managers and controllers.
OCR for page 57
Air Traffic Controller Staffing in the En Route Domain: A Review of the Federal Aviation Administration’s Task Load Model When other facility managers reviewed the PTT results, they concluded that further modifications were needed to account for even more of the D-side work that the simple conversion thresholds neglected. Accordingly, CAASD introduced the fuzzy logic modeling process. Experts were once again consulted to assign complexity weightings to the different R-side tasks and their combinations. These weightings are intended to characterize the complexity of the D-side task load, even though none of the experts consulted was asked to identify explicitly the D-side tasks involved or to estimate the time it takes to perform each. The PTT values generated from this conversion process were again presented to facility managers for feedback. Their advice led to further adjustments to the fuzzy logic inference rules and complexity weightings until the PTT values satisfied the expectations of the managers consulted. Both the conversion and validation processes involve repeated consultation with subject matter experts and facility managers and no evidence that data on the performance of D-side tasks were obtained and analyzed to assess their judgments. The heavy reliance on the experience and expectations of facility manager to evaluate the PTT estimation techniques and results is at odds with the purpose of PTT modeling; presumably this purpose is to provide independent quantitative estimates of staffing requirements. All of the PTT conversion methods applied, including the current method of fuzzy logic modeling, exhibit the same fundamental flaw—they imply an estimation of total task load without ever identifying the unmodeled tasks, much less measuring the time it takes to perform them. The conversions rely almost exclusively on experts to determine thresholds and to assign complexity weightings to the unidentified and unmodeled tasks. The D-side task loads implied by these thresholds and weightings are not validated, nor can they be in the absence of any empirical data on task performance. To adjust these conversion methods further would be insufficient and would risk making the modeling process even less transparent and less convincing. Indeed, it is by no means apparent that past adjustments have led to more accurate PTT predictions—only that they have produced values closer to the expectations of facility managers. In the case of fuzzy logic modeling, this outcome has been achieved at the cost of model transparency and credibility.